I have a series of fastq files (with up to 4000 reads in each) that I want to parse based on the time of sequencing.
So in the fastq header, date/time is listed as "start_time=2017-10-09T18:54:24z"
If I wanted to extract all sequences between 18:00 hours and 20:00 hours, is there a tool I can use to find and extract them?
4 answers
I wrote a python script now.
Execute as
python timefilt.py part1.fastq.gz --time_from 2017-10-09T18:00:00Z --time_to 2017-10-09T20:00:00Z
Well, there is not really a reason to not gzip your fastq files :)
That said, you would just have to edit this line:
for record in SeqIO.parse(gzip.open(args.fastq, 'rt'), "fastq"):
to
for record in SeqIO.parse(args.fastq, "fastq"):
Use redirect python timefilt.py part1.fastq.gz --time_from 2017-10-09T18:00:00Z --time_to 2017-10-09T20:00:00Z > new.fastq or if you want to keep the file compressed python timefilt.py part1.fastq.gz --time_from 2017-10-09T18:00:00Z --time_to 2017-10-09T20:00:00Z | gzip > new.fastq.gz
Should have known that! Thanks, you've both been a great help and I'm using it already.
I'm currently using cat to merge the fastqs to run this on multiple files, I can't think this is the most efficient method. Might there be an easy way to run this on multiple files to produce one fastq output?
Alternatively we could modify the code to make it read from stdin.
Oh not at all :) you should read from sys.stdin rather than from the file.
For inspiration you can take a look at https://github.com/wdecoster/nanofilt
Try this.
grep -A 3 --no-group-separator -E 'start_time=2017-10-09T18|start_time=2017-10-09T19' your.fastq > seq_you_want.fastq
If your files are gzipped then:
zcat your_file.fq.gz | grep -A 3 --no-group-separator -E 'start_time=2017-10-09T18|start_time=2017-10-09T19' > seq_you_want.fastq
I wrote a tool for this: https://github.com/mbhall88/ontime
You can specify your time range as a date/timestamp, or as duration from the start/end of sequencing.
For example
$ ontime --from 1h30m --to -2h in.fq
will extract reads sequenced after the first hour and a half and before the last two hours
This tool can be helpful: https://github.com/angelovangel/nanotimes
Log in to answer this question.
-
Bowtie produced empty .sam file
written by melissachua90 •
-
snpEff building new database ERROR
written by Anastasia A. •
-
Split oxford nanopore MinION fastq file based on barcode
written by ambi1999 •
-
Tophat Reporting output tracks, run out of memory, how to solve?
written by zoukai3412085 •
-
Speed up Tophat2 running
written by yuabrahamliu •
-
bcl2fastq No tiles were found to process
written by ozkanozdemir •
-
MToolBox mapExome.py error
written by Cindy •
-
Tophat 2.1.1 failing to recognize fasta file
written by kallen83 •
-
Filtering time based on time range
written by frcamacho
-
Tophat and Bowtie installed with Conda not working on test data.
written by ariel.balter
Can you post a couple of examples of full/complete headers? Is this nanopore data?
I am not aware of any tools to handle this. If you have some Python-programming experience I would be glad to help you.