This is a good point. I definitely recommend increasing max-bundle-frags above the default 500K value. However, setting it to to 10B is very excessive. We typically have RNAseq data with 1 lane of hiseq 2000 or 2500 with ~400M reads. In tests, with the default setting we see up to ~100 genes being marked as HIDATA without FPKM values calculated, some of which are genes of clear interest to your studies. Almost all of these are recovered by increasing max-bundle-frags to 10M with minimal increase in memory/runtime. Bumping up to 50M eliminates all HIDATA cases in our tests but also significantly increased runtime for reference-guided or denovo modes. In any case, there should be no reason to set it to 10B as in the case above.

Sorry for the late reply....

By position, I mean chromosomal position when Cufflinks inspects bundles. It is set to verbose, so I could see where the bottleneck was coming from. It hangs at....

Inspecting bundle 14:50232514-50232850 with 2 reads

So the next loci along I suspect has a really really deep region of reads. When I extract the reads slightly downstream of the above chromosomal position, it does have a very deep set of reads. Bundle inspection should be a fairly quick process in the Cufflinks pipeline.

I think the suggestion that max-bundle-frags needs to be reduced, is a good approach, so I'll give that a go and report back. I was just trying to avoid HI-DATA!

UPDATE: I reduced the --max-bundle-frags switch to 1000000 and after 19 hours, Cufflinks still hangs on the bundle inspection process at the same chromosomal position. This data is big (174 Million reads), but I thought the bundle inspection process was meant to be pretty quick, can anyone correct me on this? It seems that the bundle inspection does not use the --max-bundle-frags switch.

UPDATE: After speaking with a Cufflinks dev - The CURRENT (Cufflinks 2.2.0) behaviour is that enabling --GTF-guide WILL OVERRIDE the --max-bundle-frags parameter. This means that if you want to do novel discovery, there is no option to set a skip parameter for large bundles. I've been told that it's on the to-do list for implementation.

Thanks for your suggestion, but Cufflinks starts fine, it's on the bundle inspection process of Cufflinks that I seem to be getting the bottleneck. I've updated the post to give some more context.

Absolutely. I've talked (albeit very briefly) with one of the Cufflinks Devs. His suggestion was to use the -G flag as an alternative to --GTF-guide. Using -G over --GTF-guide means that you can't discover any novel isoforms or genes, however it works in conjunction with the --max-bundle-frags flag.

This is my understanding of the problem (Cufflinks 2.2.0) - - --GTF-guide Allows for the discovery of Novel Isoforms and genes but can't be used with --max-bundle-frags to streamline for extreme depth. - -G allows you to run against the reference GTF but without novel discovery. This works with --max-bundle-frags.

The mask file is good for a case by case loci but it's just not practical for the whole run.

I'm still waiting to hear back from the dev team if this a bug, or by design. I'll update the post if I hear anything.

I had a look at the thread Chris, interesting that you found the bottleneck, great job!

So from your post, I understand the problem was in the data structure of the open mates (a map of lists)? Your solution was to replace this data structure with an unordered multimap from boost, indexed with a hash of qname?

Can you explain why this caused the bottleneck in a little bit more detail? (for my own curiosity and the benefit of the original post!)

Have you tested this and seen a noticeable improvement?

Thanks for your work on this!