Is anybody has some experience with ALLPATHS-LG ?
Is that possible to run in a multi threads mod ?
Especially, the first step PrepareAllPathsInput takes me a lot of time ... It's running since 2 days, is it normal ?
The command is:
PrepareAllPathsInput\
DATA_DIR=$PWD/rep\
PLOIDY=1\
IN_GROUPS_CSV=/rep/in_groups.csv\
IN_LIBS_CSV=rep/in_libs.csv\
OVERWRITE=True\
| tee prepare.outre
•
written
by
Picasa
It is a bit tricky, but you can speed up data import by setting up a cache first with CacheLibs.pl/CacheGroups.pl. See "ALLPATHS Cache for power users", page 19 in the manual for details.
With those scripts you can add different libraries individually and in parallel, and CacheGroups.pl also allows forking of library import to parallel jobs.
Thanks for your support.
1) What do you mean by "ou can add different libraries individually and in parallel". Do I have to create a in_groups.csv and a in_libs.csv for each library and run CacheLibs.pl/CacheGroups.pl separately ?
2) "CacheGroups.pl also allows forking of library import to parallel jobs." : Is it with the "HOSTS" option ? I'm not sure how to use it (I work on a cluster)
1) Yes, that's what I did. I split up my original file in serveral groups/libs.csv and imported those.
2) As far as I understand, if you just provide a single number, it will set the number of forked processes (libraries processed in parallel) on the current node, i.e. it more or less should be equal to setting threads.
Hi,
I think you need to check if something wrong, normally the PrepareAllPathsInput will take less than two days to prepare 300G data.
The same to you. I think you need to check whether your server I/O or network have some problem.
Mon Apr 11 17:13:24 2016 (MPF): Merging. Mon Apr 11 19:41:19 2016 (MPF): Merged 816052282 read pairs. Mon Apr 11 19:41:19 2016 (MPF): Done.
This is the normal speed. If your file is writing few M per hour, then there are some problems.
I don't know, because I didn't fix, at the end I change server. I think it is something wrong with writing big file, because before MergePairedFastbs it need to convert to fastb, and the speed of writing is normal.
Around 40~45 minutes for 816052282 read pairs.
I did experience some weird problems with ALLPATHS-LG in the past on some machines that could be similar to what you are experiencing - I never really figured out the real reason or how to fix it, though. In my case, once a certain amount of RAM was loaded with data (ALLPATHS reads entire libraries in RAM), the process kind of stalled, i.e. no more I/O activity and no CPU activity, just sitting there with high memory demands. Check top or similar, to see what your jobs are doing.
OK. Let me write down what happen to my case. I am using cluster. I used ALLPATH-LG before, the whole thing is normal. Then we have more data (small data set with long insert mate pair data), and I want to rerun. At that time, the whole cluster was maintained. After the maintain, I rerun and the odd thing happen, MergePairedFastbs just output few Mb per hour. I tried different nodes, they were all the same. At the end, I have to use a local server. And, by the way, I tried three different version of ALLPATH-LG. I contacted the administrator, but he said he did not do anything to the cluster and the process is running but just very slow.
Log in to answer this question.
