blastx threading causing error

Question

[uriamo]

Hey, I'm using blast 2.2.28 and a local db (on a cluster). Whenever I tried to use multiple threads in order to speed up things but whenever I'm running the command:

blastx -db mydb  -query myreads.fasta -out file.csv -word_size 4  -num_threads 2  -outfmt 10

I get this error:

Error: NCBI C++ Exception:
"/apps/RH6U4/blast/ncbi-blast-2.2.28+-src/c++/src/corelib/ncbiobj.cpp",
line 925: Critical: ncbi::CObject::ThrowNullPointerException() -
Attempt to access NULL pointer.

Any idea what's causing it? (btw, the same command without "-num_threads" flag runs fine )

[5heikki]

Did you compile the blast binaries from source yourself? If yes, does the same occur with the pre-built binaries? Also, do you actually have 20 cores in a node?

[GenoMax]

Using 20 cores is likely causing the job to spread across multiple nodes of the cluster. For various reasons this is not a good thing (as you are finding out). As @5heikki suggested, you should limit the job to the number of cores available on a physical node. If you do have 20 cores available on a physical node then instruct your job scheduler (I am sure you are using one if you are on a cluster) to keep those 20 threads on one node.

[5heikki]

If he's not submitting the job at all but running it as in OP, then it's not going to be spread out to other nodes. He's probably running the job on head node as such (and causing immense butthurt to everyone who is using the cluster properly).

[uriamo]

I'm submitting the job to the queue by the book... Please do not make any assumptions and honor me by addressing me with the second personal pronoun.

[uriamo]

And I'm sorry for the typo but it's 2 threads

[GenoMax]

Are you including a directive to keep the two threads on the same physical node? Are you asking for 2 cores in your job scheduler command if you want to run two threads?
Do you know if this blast install is known to work previously? Was it built from source or is it using premade binaries?

[5heikki]

So how exactly do you submit it? You should include such details when making a post. Are you reserving the correct number of cores? Etc. Anyway, my guess is that this blast binary is not compiled by the NCBI and the problem is related to you/someone else compiling it wrong..

Answer 1

[uriamo]

Ok... silly me. I didn't know that and for all of those people like me who aren't used to work with Linux clusters: When submitting a multithread job to the queue, you should inform the queue that it has to allocate the proper number cores required for the job.

In our CentOs server the proper command will be:

bsub -R "rusage[mem=<MEM LIMIT>]" -n <NUMBER OF CORES> blastx -db mydb  -query myreads.fasta -out file.csv -word_size 4  -num_threads <NUMBER OF CORES>  -outfmt 10

You should also make sure that the "-num_threads" is less than or equal to the number of cores asked from the queue.

[GenoMax]

Looks like your cluster setup is strict. Normally the job may have gone through with only one thread actively running if you had asked for just one core.
Look into -R "span[hosts=1]" command option as well (if you are using LSF which seems to be the case) to keep those two threads on same physical host.