This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Forum: Arvados vs Big Data Genomics

http://bdgenomics.org/

https://arvados.org/

Has anyone tried one or both of these? Which is further along in terms of storage and retrieval of sequencing formats, especially variants?

bdg adam arvados
• 5,891 views • link
updated by Ram Ram • written by Jeremy Leipzig Jeremy Leipzig

Fresh input on the topic? Has anyone adopted the technologies?

• 0 views • link
written by podro • podro

2 answers

Hi Jeremy,

I am currently looking into both technologies and they differ from what they provide.
Arvados has basically the approach to be a platform for data sharing and more important for provenance of derived or analyzed data. This is especially very appealing if you work in a global organisation where data is distributed to several sites and de-duplication is an important topic then. This is where Arvados seems to be a very valuable platform. Even every pipeline that runs currently in a usual shell environment can be ported to ARVADOS much faster than to e.g. a Cloudera platform.

They make heavy use of DOCKER, what basically means virtualisation on applicaton level. They implement their own Map-Reduce stack and there is where it gets tricky for me. I would like to use Spark and ADAM on e.g. a Cloudera platform having the genius data provenance and de-duplication features that ARVADOS provide combined with the big and innovative community of Cloudera.

If someone in this forum can help concerning this question it would be great to get some more insight. I think using ADAM would be feasible but it is unclear to me if it would be easily possible to run SPARK on top of the data stored in ARVADOS but I am still learning and reading. In case I find an answer I would post it here.

Before you ask, I do not contribute to Arvados ;)

• 0 views • link
written by tth • tth

Hello, I'm on the Arvados team, thank you for your insightful comment.

Regarding Adam/Arvados integration, I can't share any specific plans right now, but this is something we are very interested in and hope to work on in the future.

We are also heavily involved in the common workflow language effort to standarize how tools and workflows are described so they are portable over different platforms, which will make the distinctions between underlying clustering technologies like Spark and Yarn and Arvados Crunch less relevant for day to day bioinformatics that isn't working deep in the infrastructure layer.

• 0 views • link
written by peter.amstutz peter.amstutz

I am trying Adam / Spark / BigDataGenomics for storage and retrieval / analysis on 1000 genomes VCF data.

They are hoping to have a production release end of this year.

Arvados looks like they are building everything from scratch

https://arvados.org/projects/arvados/wiki/Technical_Architecture

while Adam is building on general purpose big data infra like Spark /HDFS / parquet / YARN. My bet would be one Adam, also because Berkley AMPLab, Broad and Mount Sinai are involved in the development.

• 0 views • link
updated by Ram Ram • written by WilliamS WilliamS

Hello! Depends what you're looking for with respect to variant storage and retrieval, but I would like to note that there is already a free hosted version of Arvados that people are welcome to evaluate (you can use any google account to login).

disclaimer: I contribute to Arvados. We would definitely appreciate any feedback you have! :)

• 0 views • link
written by Nancy Ouyang Nancy Ouyang

Log in to answer this question.

More posts like this