This is a test version of Biostars. For the public version, visit https://www.biostars.org.
How to let WGCNA use multiple cores

Hi guys, I am using WGCNA to deal with some large datasets (~20000 genes on ~1000 samples). I read in the help documents that WGCNA support multi-thread to speed up calculation. But when I run functions like blockwiseModules, TOMsimilarityFromExpr, cor etc, I can set parameter nThreads, it works. But when I check CPU usages by "top", it seems all multiple threads are running on one CPU core, so I didn't gain any advantage by multi-thread (I am using a linux cluster with 24 CPU cores, and set enableWGCNAThreads() already). How can I set WGCNA to use multiple cores?

wgcna multe-thread parallel
• 9,947 views • link
updated by Kevin Blighe Kevin Blighe • written by syrttgump • syrttgump

1 answer

Can you try running these lines at the top of your script(s):

require(parallel)
require(doParallel)

cpucores <- makeCluster(detectCores(), type='PSOCK') ; 

registerDoParallel(cpucores) ; 

Sys.setenv("MC_CORES"=cpucores) ;

The last line (setting MC_CORES variable) is only for Windows OS.

Also take a look at these other WGCNA functions: https://rdrr.io/cran/WGCNA/man/allowWGCNAThreads.html

• 0 views • link
written by Kevin Blighe Kevin Blighe

I met this problem too, and using the above code could not solve the problem.

Aifu

• 1 views • link
written by afli afli

Then please show what is the error message that you are receiving. In certain situations, you will, simply, not have enough power on your computer to do the analysis that you want. You should consider reducing the size of your input data [to WGCNA]

• 1 views • link
written by Kevin Blighe Kevin Blighe

I haven't met any error, the WGCNA works well, but it only use one CPU, I've tried using enableWGCNAThreads() or allowWGCNAThreads(nThreads = 22) and so on, but have no effect. I think maybe this package just use one CPU. And I use small dataset or large dataset to WGCNA, also no change.

• 1 views • link
written by afli afli

Which operating system are you using? The parallel functions in R differ based on the 'system architecture' that you use.

• 1 views • link
written by Kevin Blighe Kevin Blighe

Maybe, I use Ubuntu 16.04 LTS, and the R package is 3.5.0 version. Can the WGCNA(r package) works with multiple threads on your system?

• 1 views • link
written by afli afli

I also use Ubuntu 16.04.

Firstly, try the enableWGCNAThreads() function, and do not use allowWGCNAThreads()

If that does not work, then first run these lines:

#Set CPU cores for parallel-related functions
  cpucores <- 16
  require(parallel)
  options("mc.cores"=cpucores)

  #Set CPU cores for doParallel-related functions
  require(doParallel)
  cores <- makeCluster(detectCores(), type='PSOCK')
  registerDoParallel(cores)

...then run enableWGCNAThreads() again.

Check if multiple threads are running via the top command in BASH. You will see the R command being forked

• 1 views • link
written by Kevin Blighe Kevin Blighe

Thanks Kevin, I try the codes above and add enableWGCNAThreads(), but this could not increase the threads number. The program has cost more than 10 hs, and is still running for just one thread. One of my friends told me that WGCNA just uses one core. Maybe there're other methods to increase the thread number. I upload the data and paste my codes as follows:

library(WGCNA)
options(stringsAsFactors = FALSE)
enableWGCNAThreads(10)
lnames=load(file="filtered_data_ready_for_WGCNA_0.5tpm.RData")
datExpr <- datExpr0
rm(datExpr0)
setwd("./maxP0.05")
power <- 7
bwnet = blockwiseModules(datExpr, maxBlockSize = 46000, power = power, networkType = "signed hybrid", TOMType = "unsigned", minModuleSize = 30, corType="bicor", corOptions = "use = 'p', maxPOutliers = 0.05", reassignThreshold = 0, mergeCutHeight = 0.15, numericLabels = TRUE, saveTOMs = TRUE, saveTOMFileBase = "Signed_hybrid_TOM-blockwise_maxP0.05", verbose = 3)

You can try this, if you could run with multiple threads please share the good news with me!

data link: https://de.cyverse.org/dl/d/1D0B60A0-D1EF-4A4F-A13E-A21901301900/filtered_data_ready_for_WGCNA_0.5tpm.RData

• 1 views • link
written by afli afli

I tried, and, indeed, that particular function does not have parallel-processing enabled - I looked through the code. I then found this, where one of the WGCNA developers states exactly this:

I notice a lot of for loops in the blockwiseModules code, which could be replaced with foreach %dopar%. I initially tried but ran into errors because the function is >3000 lines of code.

I previously 'parallelised' another much smaller functiun that also performs k-means (or other) clustering: https://github.com/kevinblighe/clusGapKB

• 1 views • link
written by Kevin Blighe Kevin Blighe

Thank you very much Kevin for your detailed reply! I accept the reality that I should use just one thread to run WGCNA at the moment.

• 1 views • link
written by afli afli

im not sure about it. this is what my initial code for wgcna 

library(WGCNA)
enableWGCNAThreads()

it works ...

• 1 views • link
written by 1769mkc 1769mkc

Do you mean that many threads work simultaneously? Could you please show your system information, and WGCNA version?

I tried again and failed again...

• 1 views • link
written by afli afli

krushnach80, good to see you again. Can you confirm that it is the blockwiseModules() function that you are using? We believe, based on This, that it is just the blockwiseModules() function that is not yet multi-thread enabled.

I imagine that you are using a different WGCNA function.

• 1 views • link
written by Kevin Blighe Kevin Blighe

@Kevin sorry for the late response yes im not using "blockwiseModules()" .

• 1 views • link
written by 1769mkc 1769mkc

Log in to answer this question.

More posts like this