problem about parallel in R

Dear all: I have a question when I run a R script in parallel.

#
library(parallel)
library(mvtnorm)
source('./function.R')

snp <- read.table('./high_19.gt', header = T, check.names = F)
pheno <- read.table('./area_h.txt', header = T, stringsAsFactors = F)#, 
                    #colClasses = c('character', 'numeric', 'numeric', 'character', 'character'))

par0 <- c(172.6193597,  94.0255600,   0.1237499,   1.08,   0.8)

cl <- makeCluster(7)

clusterEvalQ(cl, library(mvtnorm))

clusterExport(cl, c('par0', 'snp', 'pheno'))

clusterEvalQ(cl, source('./function.R'))

res <- parLapply(cl, 1:1000, 'mle')    

stopCluster(cl)

###########

My snp file has 10000+ rows.

When I set parLapply(cl, 1:1000, 'mle'), it works.

However, when I set parLapply(cl, 1:10000, 'mle') or more, it will report error like this

Error in checkForRemoteErrors(val) :
  7 nodes produced errors; first error: length of 'dimnames' [1] not equal to array extent

It seems like some parameters are not used in several nodes.

Why it couldn't run when the list was set too large? What should I do to solve this problem?

Thank you

• 3,101 views • link

updated 6.6 years ago by Kevin Blighe

• written 6.6 years ago by Chenfei Zheng

You are using Windows, I assume? Otherwise, mclapply() is a better option.

On Windows, can you try to register correctly the cores and parallelisation env. with:

registerDoParallel(cl)
registerDoSEQ()

What are your system specifications?; what is your sessionInfo()?

• 0 views • link

written 6.6 years ago by Kevin Blighe

Yes i'm using windows.

Here is my sessionInfo()

R version 3.5.1 (2018-07-02)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows >= 8 x64 (build 9200)

Matrix products: default

locale:
[1] LC_COLLATE=Chinese (Simplified)_China.936  LC_CTYPE=Chinese (Simplified)_China.936    LC_MONETARY=Chinese (Simplified)_China.936
[4] LC_NUMERIC=C                               LC_TIME=Chinese (Simplified)_China.936    

attached base packages:
[1] parallel  stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] mvtnorm_1.0-10

loaded via a namespace (and not attached):
[1] colorspace_1.4-1 compiler_3.5.1   tools_3.5.1      Rcpp_1.0.1

I have tried to run this parallel in a Linux system, the same issue occurs.

I have no idea how to transfer all my parameters and functions into each node by using mclapply().

Could you help me with more detail about mclapply()

Thank you

• 0 views • link

updated 6.6 years ago by Kevin Blighe

• written 6.6 years ago by Chenfei Zheng

0 answers

No answers yet.