Best compression for single cell RNA-seq object
I generated a scRNA-seq object (counts, PCA, UMAP embeddings, DEGs etc.) in Scanpy or Seurat. What is the best data structure to store this in to reduce the size of the object?
I'm considering H5AD (scanpy/anndata), RDS or H5Seurat (Seurat), or Loom
Fast loading/access would also be good of course, thanks
• 4,698 views
•
link
updated
by
chansigit •
•
written
by
MutationalMeltdown
•
written
by
MutationalMeltdown
0 answers
No answers yet.
Log in to answer this question.
More posts like this
-
Discrepancy in gene expression between cellranger and velocyto pipeline
written by sarang.chafle •
-
Tutorial: Single-cell RNA-seq: Preprocessing: Preprocessing the data of scRNA-seq with omicverse-1
written by Julia Ma •
-
SoupX with scDblFinder, matrix error
written by sushisena •
-
h5ad cellxgene to R
written by firestar
-
scRNAseq analysis - h5ad file conversion to Seurat format
written by isd1989 •
-
scvelo Velocity plots raise KeyError(f"None of [{key}] are in the [{axis_name}]")
written by ncp2306 •
-
Seurat object
written by lsudupe •
-
VisCello Shiny app code for hosting snRNA-seq data
written by LacquerHed •
-
Produce separate matrices from clustering expression data using scanpy
written by el24 •
-
PCA vs ICA prior to t-SNE or UMAP
written by rtrende •
For archival purposes? or for access in other tools? gzip fast to decompress on the fly as well.
To access, imagine a database of objects. H5AD supports gzip compression and I use it. The main issue is the counts matrix itself (even a sparse matrix tends to be far larger than the annotations). All the approaches I suggest (apart from RDS) use HDF5 format, which seems pretty optimised, but I'm interested in the differences between them and any alternatives
I don't fully understand the problem and its requirements, but gut feeling wise I would steer far-far away from RDS, and instead would design it by relying on sparse matrix save with scipy/numpy (scipy.sparse.save_npz) then model the rest of the information as a relational database in SQlite.
I feel that would give the highest level of flexibility and extensibility for the future.
Right, but that's similar what the anndata (the object written to H5AD) is already - isn't that reinventing the wheel or am I missing something? https://anndata.readthedocs.io/en/latest/
if it is H5AD then can't be a relational database, right?
I honestly think that storing biological data in hdf5 format is a mistake, relational databases are more elegant, robust and simple to use from any language or no language at all,
the problem with relational database is the scalability and dimensionality is beyond relational database's limits. for mysql, only hundreds of numerical columns could be hold. for oracle, 1k column is its upper limit. We are building a specialized database called unified giant table holding the large scale omics data.