This is a test version of Biostars. For the public version, visit https://www.biostars.org.
How determine to read specific size from file using R language

need help? , my problem is that i have fastq file with size 50 Gigabyte i cannot read it using R because my ram is 8 giga, so i want to read specific size from it and make it as stream to read x size until i finish operation on all the 50 giga !

r fastq rna-seq python
• 2,210 views • link
updated by jomo018 jomo018 • written by emadelhewihy • emadelhewihy

You could split the original file into smaller chunks.

• 0 views • link
written by GenoMax GenoMax

how i do that using R !? is there function to do that ?

• 0 views • link
written by emadelhewihy • emadelhewihy

You could use reformat.sh from BBMap suite to break the files into smaller chunks. reformat.sh will work on Windows/Unix as long as you have Java installed. Look at the "sampling" options in in-line help for reformat.sh.

• 0 views • link
written by GenoMax GenoMax

Not sure why your question is about R and you also tag Python but in Python you could just iterate over the file without loading it into memory first. I don't know about R though.

• 0 views • link
written by WouterDeCoster WouterDeCoster

i also tagged python to know how to do that in case of i did not found that by R, so is there link to do that by python?

• 0 views • link
written by emadelhewihy • emadelhewihy

1 answer

You can use read.table() with nrows and skip tags.

• 0 views • link
written by jomo018 jomo018

the file is Fastq extention so it does not have specific schema like TSV or CSV it look like as string

• 0 views • link
written by emadelhewihy • emadelhewihy

jomo018 just gave you the solution. No need to explain how a fastq file looks like.

• 0 views • link
written by WouterDeCoster WouterDeCoster

Log in to answer this question.

More posts like this