This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Can clumpify.sh be used in a streaming fashion downstream of bcl-convert

Hi

I wanted to know if it is possible to use clumpify.sh in a streaming fashion - like one would do with bwa mem .... | samtools

I would like to demultiplex with bcl-convert and on the fly, I want to use clumpify.sh.

Thanks in advance.

bbmap clumpify bclconvert
• 1,998 views • link
updated by Brian Bushnell Brian Bushnell • written by tamu.anand • tamu.anand

1 answer

I don't think you can do that though @Brian will perhaps provide an official word. Limitation may be with bcl-convert. bcl-convert writes output to files (not sure if you can make to write to STDOUT). Then there will be the issue of paired-end data. clumpify.sh will read from STDIN (which I tested) but the data may need to be interleaved when streaming since there is only one STDIN.

• 0 views • link
written by GenoMax GenoMax

Thanks @genomax. I will await Brian's guidance on this.

• 0 views • link
written by tamu.anand • tamu.anand

This is correct. Clumpify will accept stdin and work fine as long as all the data fits in memory, but it doesn't fully stream since it needs to load every single read before it can start deduplicating or sorting.

For example:

bcl-convert | clumpify.sh in=stdin.fq interleaved out=clumped.fq.gz

...that assumes input is interleaved and I don't know whether bcl-convert does that, but it might, since I think all of our files come out of demultiplexing already interleaved...

There's also the issue of multiple output streams... I don't personally run the bcl tools but if you had multiple output streams at once, I don't know how you'd pipe them all. One at a time is fine though.

• 0 views • link
written by Brian Bushnell Brian Bushnell

since I think all of our files come out of demultiplexing already interleaved...

Only if JGI uses non-Illumina software to convert basecalls.

• 0 views • link
written by GenoMax GenoMax

Nope, we use Illumina software, so I guess there's a downstream process for interleaving them before they get archived. It makes tracking a lot easier when one library yields one file.

• 0 views • link
written by Brian Bushnell Brian Bushnell

Log in to answer this question.

More posts like this