This is a test version of Biostars. For the public version, visit https://www.biostars.org.
How to plot a histogram for billions of genotype quality values?

How can I best plot a histogram for billions of genotype quality values?

I have a simple one column file with billions of genotype quality values. The file is several GB uncompressed.

Is there a statistics library in Python or R that can build up a histogram by streaming trough the data? Instead of loading everything to memory and then creating the histogram? I prefer using all of the data versus sampling it.

Or do I have to write a script first to collect the counts per bin and then give those count per bin to R for plotting? This functionality feels like it should already exist in a stats library some where.

I know the min and max of the values and would be able to specify a bin size.

vcf quality r python
• 3,860 views • link
updated by biocyberman biocyberman • written by William William

what kind of graph do you need ? qual=f(pos) ?

• 1 views • link
written by Pierre Lindenbaum Pierre Lindenbaum

I prefer using all of the data versus sampling it

Could you motivate this? If you want to produce a histogram to visualize the distribution of quality values, I don't see why you need billions of data points, especially since the range of quality values is discrete and not very large. After you have collected a million or so data points at random you have a pretty good estimate of the all thing.

• 2 views • link
written by dariober dariober

Sampling also adds complexity and risk doing it wrong. I at least would need to think about how to do it correctly. By itself the frequency of the analysis and size of the data still allow for processing the total collection.

• 1 views • link
written by William William

2 answers

Or do I have to write a script first to collect the counts per bin and then give those count per bin to R for plotting? This functionality feels like it should already exist in a stats library some where.

This is what I'd do. You're right that it might exist, but writing a 10 line perl script to bin them might be quicker than searching. I'd wager you'd be done before someone chimes in with the library to do it :)

• 1 views • link
written by Chris Miller Chris Miller

Meet the selling point of Datashader: http://datashader.readthedocs.org/en/latest (even though I tend to filter and subset the data whenever possible).

• 2 views • link
written by biocyberman biocyberman

Log in to answer this question.

More posts like this