BBduk reading fastq from S3 directly - Is it possibile?
Hello to all,
I am not from Bioinf field but there is no issue for me running bbduk trimming command :)
I was wondering is it possibile to load paired fastq reads directly form S3 bucket?
Even better, load input from S3 and stream the output to S3 directly as well = ]
I find many tools are "missing" this feature now that the cloud is "default" go to. Especially for storage.
0 answers
No answers yet.
Log in to answer this question.
More posts like this
-
How to analyze SRA metagenomic data without downloading large files locally (Galaxy or cloud altern…
written by usalamah718 •
-
Converting SRA Data to FASTQ and Saving Directly to S3
written by Soohyun •
-
Accessing SRA file with multiple fastqs under one accession
written by tony_88888 •
-
Fetch Fastq files directly for SRA data
written by Chintan •
-
bbduk command line options for EM-seq (NEB)
written by tamu.anand •
-
How to download NCBI fastq data to local machine where access type is "Use Cloud Data Delivery"
written by daewowo •
-
Copying BCL files from BaseSpace directly into tar.gz
written by GBC_Zonatos •
-
quality trimming with trimmomatic
written by blooming.daisy333 •
-
Individual Hit Gene Information From Qblast Xml Output
written by Jorge •
-
How Much Does It Cost To Align A Flowcell In The Cloud?
written by Jeremy Leipzig
I think this is because people prioritize solving problems that are objectively more difficult. Storage is not an issue for most people in this field, as one can buy a 4-10 Tb hard disk for well under $500.
Yes ofcorse. I did not mean to criticize. I am aware that my correspondence is blunt sometimes though :).
You can use a workflow orchestrator like nextflow - make sure to configure it with your aws stuff and then use nextflow's s3 storage support to write your workflow(s) - this way you can stream data right from S3 and write back to s3.
IMHO, using the fusion filesystem with Nextflow will be best as shown here
Hi, thanks for taking time to share. Since I don't come from bioinf side my question probably lacks context. Question is more of a "low level" nature. Lie techincal without using "third party" tools. Rather I asked it for educational purpose.
I will ask similar in a new thread hoping to make myself more clear.