Hi, I have a really large BLAST XML file - something like 30gb in size.
I'd like to reduce it so I can run through it quicker with Biopython.
Is there a way to reduce the file by keeping something like the top 25 hits based on bitscore for each query.
Preferably I'd like to do with Python/Biopython.
Thanks
4 answers
To extract a subpart of a large XML file, the idea is to use a XML pull parser. Read and echo each node of your XML until you find a [?]Hit[?] element. Then, parse but only echo the blocks of XML elements you need.
In java, a Pull parser specific to a given DTD can be generated with xjc.
For an example, see A: How To Retrieve Human Proteins Sequence Containing A Given Domain .
If your document can fit in memory, you can use XSLT:
see Standalone Blast Options or How To Filter A Blast Xml Output ?
•
written
by
Pierre Lindenbaum
You could compress the file using something like parallel bzip then read from the file as a compressed IO pipe. Obviously this only reduces the physical size of the file not its contents. You'll also expect a longer running time too I imagine.
Someone else had a similar question at about the same time as you. This looks like a good answer: A: Can Biopython Parse Gzipped Xml From Blast?
import gzip
from Bio.Blast import NCBIXML
blast_file = gzip.open('output.xml.gz', 'rb')
blast_records = NCBIXML.parse(blast_file)
The file size isn't so much an issue but the time it takes to iterate through the files that's annoying.
The whole reason I'm trying to do this is to get the full description of the hit. My solution now has been to directly parse the human readable default blast out to tab delimited.
These are much smaller files and parse quickly.
Thanks for the thoughts though.
Log in to answer this question.
-
Blast-XML and Biopython: extract query-anchored results (i.e. MSA)
written by LeWöps •
-
blastp short in BLAST+ vs. BLAST web interface
written by smiller
-
Nucleotide CDS from RefSeq?
written by Prohan
-
How To Extract Desire Genes Blast Xml Result From A Big Blast Xml File
written by KJ Lim •
-
[Help]: Extracting Sequence From Fasta Using Blast Informations?
written by francois.maginiot •
-
Get Snps, Indels From Biopython Blast Parser
written by Richard
-
Beginner Blasting Short Sequences
written by Richard
-
Can Biopython Parse Gzipped Xml From Blast?
written by Karthik •
-
Parsing Ncbi Taxonomic Tree?
written by Prohan
-
Getting Top 10 Sequences Of Blast Results Bio Python
written by Ankur •
It would be more sensible to generate a smaller XML file in the first instance, using BLAST command-line parameters. But perhaps you did not generate the original file?
Currently Biopython parses BLAST XML, but doesn't write it out again. That would be the most elegant Biopython-based solution to your needs.
Otherwise you'll need to write something specific, possibly using one of the built in Python XML libraries, or hand coded for this specific need.
Yes I next time I'll reduce the number of reported HSPs. I was trying to be smart by keeping everything and didn't think parsing would be a problem later.
If you are interested in parsing some information from the big XML file you can use the event parsing strategy. lxml in Python has great event parser. Please drop a line after this comment if you are interested in this approach and I will post a detailed methodology.
@Bio_Neo: Could you describe this in more detail? I have a BLAST result with top 50 hits of a large multifasta in XML format. I need to reduce this to top 5 hits for using for some downstream processes. Could you explain how to use the event parsing strategy for this?