I have two list of lists A and B, A and B contain 100 data frames each and the dimension of each data frame is 15000 X 15000. I would like to find the correlation for the entire data frame in the following way: Consider the first list in both lists and find cor (A,B) and get a single value correlating the entire dataframe. Similarly consider the second list in both lists and find cor(A,B) and continue this for the 100 dataframes.

I tried the following:

  A # list of 100 dataframes
  B #list of 100 dataframes

  C<- A[1] # extract only the first list from A
  D<- B[1] # extract only the first list from B

  C<-unlist(C) ### unlist C
  D<-unlist(D) ## unlist D

Then computed

   Correlation<- cor(C,D) ## to obtain a single correlation coefficient to see how these two vectors are correlated

But I end up with the error sayin

  R cannot allocate a vector of size 3.9 GB

Is there a better way to do this in faster way which could be implemented to the entire list. I work on a server which allows me to compute large values but it still shows up this error and the unlisting takes ages because of the size of the dataframe.