This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Quick Command To Split Large Blastx Files?

Any thoughts? I have large 7 gb files as resulted from Blastx any clue how to split them?

Edit:

These are standard blastX files (fasta). I need to split them for a program (binning) call SOrt-ITEMS/MEGAN

blast

what's the format of your file (xml ?...). Why do you need to split them ?

By "fasta", I assume you mean the format of the input sequences for blastx (which is always fasta), not the output.

I'm tempted to close this as "no longer relevant", since the author has not provided further input (votes, clarification) for 6 days.

I'm closing this as "no longer relevant", since the author has not provided further input (votes, clarification) for 6 days.

4 answers

It depends on your output. BLASTXML can be parsed with something like the perl module XML::Merge, but this would consume a lot of memory and take a while, so I would recommend grabbing what is essentially the header and the footer of each file (the statistics at the end will be thrown off, but who cares?) and cutting the meat into chunks. If it's pairwise you could write a script to delimit the file on a regular expression of the GI. If it's tabular just copy out chunks of lines with something like sed or head/tail. If it's tabular with comments use something like below:

wc -l filename #determine the number of lines. Modulus (% operator) by 2 for a single line comment
split -l numberOfLines filename newfilename #new file will iterate, producing newfilename0 newfilename1...

I think the question is about splitting BLAST result files, not splitting sequences prior to BLAST.

I thought bitscores were comparable, but perhaps you are thinking of e-values, which depend on database size?

Ketil, I was fairly sure I read they weren't comparable, but I may be wrong. I skimmed through where I thought I read it and couldn't find anything on it and didn't feel like doing the math/logic to see if they are, haha.

I made this script to split files into X pieces. If your results are on a line each this should work, and if they aren't then it can be modified fairly easy.

Good luck.

#!/bin/bash

fN=$1
outD=$2
numParts=$3
bn=`basename "$fN"`

echo using basename $bn
#get the number of lines in the file
IFS=' ' read -a ARR <<< `wc -l $fN`
numLines=${ARR[0]}

#Get lines per part
#echo $numLines
lpp=$(($numLines/$numParts + 1))
#echo $lpp

#make outdirectory if it doesn't exist
if [ ! -d $outD ] ; then
    mkdir $outD
fi

#split file into X number of parts via lines
echo splitting file into parts with ending: ${outD}/${bn}.splitted.
split -l $lpp -d $fN ${outD}/${bn}.splitted

These are standard blastX files (fasta).

I need to split them for a program (binning) call SOrt-ITEMS/MEGAN

please don't put extra information as an answer, edit your original question and insert this extra information there (please do this within a day as this answer will be deleted)

Splitting BLAST output in the default format is a little tricky, because you may create files that are not recognised as valid BLAST output by other software.

In BLAST output, each new result has a first line that begins with "Query=". So you could use csplit like this, where FILE is the BLAST output:

csplit FILE /Query=/

This will split the output into separate results for each query sequence. It will also generate the same number of identical "header" files, which will have the same size (around 481 bytes) and look something like this:

BLASTX 2.2.25+
Reference: Stephen F. Altschul, Thomas L. Madden, Alejandro
A. Schaffer, Jinghui Zhang, Zheng Zhang, Webb Miller, and
David J. Lipman (1997), "Gapped BLAST and PSI-BLAST: a new
generation of protein database search programs", Nucleic
Acids Res. 25:3389-3402.

RID: VDNP132J01S

Database: All non-redundant GenBank CDS
translations+PDB+SwissProt+PIR+PRF excluding environmental samples
from WGS projects
       13,811,796 sequences; 4,741,152,926 total letters

Note: that comes from a web BLAST at NCBI, so local standalone BLAST will be slightly different.

You may need to concatenate a header to each result in order to generate a new valid BLAST output file. Here's an example for one result file; you'd want to use a loop or some other shell tool for all result files:

cat xx00 xx01 > xx01

Having said all that: the SOrt-ITEMS documentation states that BLASTX output "is then taken as the input." So it may be that SOrt-ITEMS will not work correctly if you split the BLAST output. The only solution to that is run SOrt-ITEMS on a machine with lots of RAM.

Log in to answer this question.