I think we always could come arcoss large files, say, fastq files. Now I have really super large fastq files, around 10GB. I need to first split it into some smaller files. I've written some python script to do this, with the algorithm of taking the whole file into memory (similar to readlines(), or zcat file), thus leading to insufficiency of memory (60GB memory on cluster node, but still not enough)
Just wondering, is there any algorithm which doesn't take the whole file, but read line by line? Anyone wanna share any script for splitting?
BTW, I'm not doing BWA paired-end mapping; but curious is it possible to run BWA with an input file with size around 10GB? thx
Actually I'm using python to split. One of the key command here is:
input = commands.getoutput('zcat ' + fastqfile).splitlines(True)
Seems a bit faster than readlines(); but basically the idea is still to create "list" or in perl called "array". Then I can manipulate specific line of the list, say list[1000] (the 1000th line)
•
written
by
Bioscientist
3 answers
I agree with Wen.Huang that it's not that large, but it will help parallelization if you split.
You definitely do not need to read the entire file into memory. There's a unix command, split that does what you want. Here's an example.
# make a fake example file.
for i in `seq 10000`; do echo $i >> t.fake; done
# should be a multipe of 4 for fastq
LINES_PER_FILE=1000
split -l $LINES_PER_FILE t.fake output_prefix
ls output_prefix*
Will create:
output_prefixaa output_prefixac output_prefixae output_prefixag output_prefixai
output_prefixab output_prefixad output_prefixaf output_prefixah output_prefixaj
A small perl code as an example:
## Split the file into 10MB pieces for sorting
system("split -b 10k $file_path$output_file $output_dir$split_file_name");
## Split the merid into 10000 lines pieces for sorting
system("split -l 100000 $file_path$output_file $output_dir$split_file_name");
hope this helps.
In perl you can do it in 2 ways:
## Split the file into 10MB pieces for sorting
system("split -b 10k $file_path$output_file $output_dir$split_file_name");
## Split the merid into 10000 lines pieces for sorting
system("split -l 100000 $file_path$output_file $output_dir$split_file_name");
hope this helps.
thanks. BUt I'm kind of compressed files....for which I cannot use "split" commands.
great thanks! works super!
csplit example: http://41j.com/blog/2011/01/split-fasta-file-into-files-with-one-contig-per-file/ csplit -z myfile.fa '/>/' '{*}'
A small perl example of what Brentp mentioned:
## Split the file into 10MB pieces for sorting
system("split -b 10k $file_path$output_file $output_dir$split_file_name");
## Split the merid into 10000 lines pieces for sorting
system("split -l 100000 $file_path$output_file $output_dir$split_file_name");
hope this helps.
this is not really a Perl example, it's a shell example being called by perl ...
Log in to answer this question.
-
Any Quick Sort Method For Large Bed File As 20G In Size?
written by Xianjun
-
Comparing Text Files Using Python
written by Nitin
-
Error When Running Fastx (Fastx_Quality_Stats)
written by neal.platt
-
Gatk Multi-Sample Calling
written by Bioscientist
-
Softwares For Simple Statistics Of Bed Files
written by Bioscientist
-
Extract One-End Mapped Paired-End Reads From Bwa Bam/Sam Files.
written by Bioscientist
-
Root File Output Problem
written by Bioscientist
-
Sequence Number Count In Fastq.Gz File
written by Bioscientist
-
Use Python To List Filename With Specific Extensions.
written by Bioscientist
-
Run Hundreds Of Bwa Commands Without Waiting
written by Bioscientist
I've successfully run BWA with files over 20 GB (compressed) in size.
Also, what are you trying to do? 60GB is more than enough for most alignment needs unless you have a really large genome, whereas it may fall short for assembly and splitting your reads would not help much here.
I've successfully run BWA on compressed fastq files over 20GB in size (around 40 GB uncompressed) on a machine with 18GB of RAM.
python has a gzip module--no need to the zcat call.