This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Quantiles Normalization On Massive Dataset

Hi,

I've got a massive dataset of 160 Affy Exon arrays that I need to quantiles normalize at probe level (the reasons for which are complicated!), which means ~6 million probes per sample.

Trying to load the into memory maxes out 32gigs of memory on one of our cluster nodes.

I clearly need to find a way to do this without loading it into memory, would anyone perhaps have a pointer/suggestion?

Thanks!

normalization
• 3,105 views • link
updated by Biostar Biostar • written by Paul Paul

2 answers

A few suggestions:

  • Aroma affy or xps for on-disk access instead of loading everything into memory,
  • fRMA (frozen RMA) to pre-process single or batches of arrays against precomputed reference probe effects.

Hope this helps.

• 1 views • link
written by Laurent Laurent

I also recommend Aroma affy, but you might also consider RMAExpress

• 0 views • link
written by Malachi Griffith Malachi Griffith

Log in to answer this question.

More posts like this