Hi,
I have a continuous variable A which is not parse, and I have a group of continuous variables which are very sparse (some of them have only one non-0 value). I want to calculate the correlations between variable A vs each of the variable in the group. I used cor.test() from r package stats, in which the default test is Pearson test. However the results look not very trustable. One variable that has only one non-0 value shows the most significant correlation with the variable A based on the p value.
I wondered if I'm using the wrong test on this type of data? What is a better way to calculate their corelations?
Thanks!
•
written
by
leranwangcs •
Spearman doesn't work well with sparsity -- it is based on ranking and if you have a bunch of zeroes, it's hard to rank. Kendall tau works better for a nonparametric approach I think.
The issue doesn't appear to be because of linearity, it appears to be because of sparsity.
Distance metrics are nice for measuring associations. If you look at the formula for R^2, it is actually a standardized version of the MSE, so I might suggest trying out different distance metrics.
Log in to answer this question.
Hmm, perhaps try a distance metric like mean squared deviation?
Thanks for the suggestion! Could you please give me some more details on how to do this?
Thanks so much!
You will need truncated SVD for sparse data. Have your data matrix, select the number of components (I suggest 5-10), and that is pretty much it.
https://scikit-learn.org/stable/modules/generated/sklearn.decomposition.TruncatedSVD.html
Not sure if this has foundation in statistics.
I suggest you try doing a singular value decomposition on both datasets, then take the first 10 components and calculate the correlations of those vectors.