Thank you for the reply.
Hello,
I am running the following script in a Linux-based cluster.
for FILE in $FASTQ_DIR/*.fastq
do
BASENAME=$(basename $FILE .fastq)
bwa mem -t 4 -M $REFERENCE $FILE > $BAM_DIR/$BASENAME.sam
done
When I check the status after running this code, it shows that I am using only 1 CPU and 0.4% of the memory. The cluster has 256 CPUs and 512 Gb RAM.
Is there a way to specify more number of CPUs and RAM to be used for specific codes? I do not have the SLURM system installed in the cluster.
Previously I used the following code, but it did not run or do anything.
systemd-run --scope -p CPUQuota=60% -p MemoryMax=300M --user ./test.bash
Thank you
2 answers
By specifying -t 4 in the bwa command line, you are telling bwa to use 4 threads. Now, each of those threads will not neccessary max out a CPU, because at any one time, each thread might be waiting for data from disk, or waiting for another thread to finish what it is doing to provide it with the information it needs.
If you want bwa to use more CPU at once, you need to increase the number of threads it is allocated by using a higher number than 4 in your -t option. However, be aware that generally there is a point of dimishing returns on allocating more an more threads to a single task, where adding more threads doesn't help becuase you are bottlenecked by something else.
bwa will use the memory it uses - in the case of batch linux apps like this, allocating more or less memory makes no difference to speed as long as memory is sufficient that there is no swapping to disk (large linux servers usually have disk swapping disabled anyway).
However, the best way to speed this up would be to run multiple instances of bwa at once. At the moment you are running on one fastq, waiting for that to finish, and then running on the next. You would improve things by running 4 or 5 instances at once. You can do this with the parallel tool https://www.gnu.org/software/parallel/.
Presumably your system has 512 Gb of memory, because 512 Mb would be tiny. So if you are specifying 300 Mb on command line, that would be about 0.4% of memory.
Log in to answer this question.
read the doc :https://blog.ronin.cloud/slurm-intro/
There is no SLURM in the cluster I am working on.
Oops, my bad: I saw the keyword slurm on OP's title and added it as a tag. OP mentions "without SLURM"
You must mean 512 GB of RAM.
If you are working on a cluster then presumably it has some kind of job scheduling system. This question is best addressed to your local systems tech support.
Ah sorry. Yes, it is 512 GB of RAM. I addressed this and got a reply that they are working on building a job scheduling system. So at the moment, there is no such scheduling option on this cluster.
Looks like you already know how to limit system resources (
systemd-run) based on your original post.Please keep in mind that aligners may not use all cores/threads at all times. As long as you are specifying the right options things should work.
You deleted a bunch of posts that had all received feedback. Please do not do this again, or your account will be suspended.