This is a test version of Biostars. For the public version, visit https://www.biostars.org.
How to use whole computer cluster power for computation of a single job?

I know the computations by breaking the files and parallelize it in SGE or SLURM. I want to use whole computation power (20 node , 80 cores) for a single job without breaking it. If I just submit it (a single perl script for large computation by AutoFact program) from master node it is occupying the cluster but not computing with 20 node and 80 cores.

blast perl sequence alignment
• 2,494 views • link
updated by Alex Reynolds Alex Reynolds • written by sammy.ich17 • sammy.ich17

Hi,

I have a feeling more details (specifics on what you're trying to accomplish, what the perl script does, etc) will help you get an answer that is best suited to your scenario. The question in its current form is pretty vague and offers little detail on your *exact* current procedure. Pl help us help you better :)

• 0 views • link
written by Ram Ram

"for a single job without breaking it" seems incompatible with "computing with 20 node and 80 cores."

• 0 views • link
written by Pierre Lindenbaum Pierre Lindenbaum
My question though is what is the "it" here? Breaking the input file or breaking as in damaging the process? OP needs to be more specific and unambiguous IMHO.
• 0 views • link
written by Ram Ram

2 answers

You'd have to rewrite your perl script to use MPI so that you can dole out computations between the nodes. Just running a program on a cluster doesn't make it parallel, you have to design it to use the resources.

• 0 views • link
written by Devon Ryan Devon Ryan

Looking at the AutoFACT scripts, it's not clear that MPI will help. Looking over the methodology, it looks like it just does a bunch of searches to various databases, correct? If it's not a computationally intensive process for AutoFACT, that suggests a parallelization approach, where each of various searches can be set on their own node. Take a look at the Schedule::SGE module and consider modifying the AutoFACT code to support scheduling jobs, where jobs are database searches.

• 0 views • link
written by Alex Reynolds Alex Reynolds

Log in to answer this question.

More posts like this