This is a test version of Biostars. For the public version, visit https://www.biostars.org.
How do you know if a sequencing dataset is safe to run before pipelines?

I’ve been running into situations where FASTQ files pass QC individually, but downstream pipelines (alignment, variant calling) still fail or give poor results.

This makes me wonder if per-file QC is enough, or if something is getting missed at the dataset level.

Do people have a standard way to determine if an entire dataset is actually “safe to run” before kicking off expensive pipelines?

Or is it mostly trial and error after running alignment/analysis?

Curious how others approach this in practice.

sequencing ngs quality-control fastq pipelines

3 answers

Do people have a standard way to determine if an entire dataset is actually “safe to run” before kicking off expensive pipelines?

Sequenced samples may "fail" during the analysis for various reasons. These may be related to analysis program choice, options used, design of experiment and suitability of the data for the particular analysis. A sample passing gross QC is not guaranteed to always generate usable data (e.g. DNA contamination in an RNAseq experiment, failed ChIP etc can't be reliably detected until some analysis has been done).

In general, if you are generating your own data (collecting samples, generating libraries) then you have much better control over the entire process. So by the time you get to the sequenced data, you should have a good idea of how that data is likely to behave.

On other hand, if you are using public data from sources like SRA then it can be a bit of a "wild west" situation for some fraction of data. This may be more prevalent in older datasets that are not of quality we have come to expect of current datasets.

At the end of the day, analysis of sequence data is an experiment. Sometimes you may end up with a negative result, even if everything has been carefully controlled. Negative results may be informative in instances but they will always end up having a cost (money and effort).

There should be a log after you finished the pipeline, so checking the error code may could help you locate issues and gain more details

QC is there to validate that your input data is technically sound, not to 'indicate' that it will answer your research question nor that it might be the right dataset to do an analysis with. Those are two different 'goals'

out of curiosity: which kind of pipelines are you thinking of that you find 'expensive'? we're not talking biocomp related then?

Log in to answer this question.