Hi all,
Could I please know if there is quickest way to extract reads from a huge fastq file to another. I already tried the following.
grep -A3 '1:N:0:' ORGAN1.fastq >ORGAN1.cleaned.fastq
but grep takes too long. Any oneliners from you are very much appreciated.
Thank you
Prakki Rama.
•
written
by
Prakki Rama
A faster way is to do this:
LC_ALL=C fgrep -A3 '1:N:0:' ORGAN1.fastq >ORGAN1.cleaned.fastq
Here is an explanation of why this is so much faster.
•
written
by
Philipp Bayer
Wow!! Normal grep on a sample file took 17 sec, whereas LC_ALL=C just took only 4 sec. Wonderful! Thank you very much.
•
written
by
Prakki Rama
For finding the fixed string using LC_ALL=C fgrep is very fast. But when it comes to finding regex, it is slower (although slightly faster than normal grep).
•
written
by
Prakki Rama
fgrep doesn't work with regexes (that's why it's faster), could it be that it switches to egrep or grep -g for you?
Yes. You are true. It does not work for regex. I was only looking only at the time of execution. My mistake.
If you are repeatedly querying this file, try splitting it into smaller units (say, with UNIX split), and then search through the smaller files in parallel. You could do this with, say, jobs scheduled on an SGE grid, or with GNU Parallel.
Thank you Alex. But, I might not need repeatedly query the file. Grep is taking long time. sed's situation is also more or less seems same.
Try to avoid adding an answer if you're not answering the question. You can always use the "Add Comment" button below the question or below another answer if you want to.
Log in to answer this question.
