This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Optimal K-means cluster # for large data (R)

Background: I have multiple data matrices, each with 2,000 columns and ~21,0000 rows. I am performing a K-means analysis and then producing a heatmap of the cluster data. I am working in R.

Problem: Rather than just pre-selecting a K-means cluster number and through trial and error choosing which plot "looks best", I am trying to use a tool that will perform something like the elbow or silhouette methods to determine optimal cluster number. I have tried nclust (prior to implementing nclust, I have used the amap package to calculate the distance matrix). My problem is that after ~5 hours it doesn't finish running. I receive no errors or warnings. I'm remoting into a server for this data; eventually I lose connection anyway so I can't wait many hours, besides the practical consideration.

Question: Is there a practical solution or tool that can handle a large matrix for determining optimal cluster # for a k-means analysis?

r kmeans
• 3,794 views • link
written by crcarroll • crcarroll

1 answer

What about gap statistic. clusgap() function of cluster package. I think in single cell data, you first obtain PCA and then use those reduced dimension for k-mean clustering. Look at the below link, this is for single cell, but I am sure can be applied to your problem.

https://osca.bioconductor.org/clustering.html

• 0 views • link
written by piyushjo piyushjo

Thank you, I will look into this.

• 0 views • link
written by crcarroll • crcarroll

Actually that is my question on bioconductor. I wanted to ask Aaron about this stuff as it was related to his scran package. The OP here was just quicker than me, however, this question was still about k-means and I was more interested in optimizing graph based clustering. Sorry if it felt like a double post!

• 0 views • link
written by piyushjo piyushjo

No, no problem. The question in this thread here is from crcarroll. As I mentioned in my own answer on Bioconductor, there's no right or wrong answer. Aaron's knowledge definitely supersedes mine in this area though.

• 0 views • link
written by Kevin Blighe Kevin Blighe

Log in to answer this question.

More posts like this