This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Using tabix remotely against S3

Hi everyone,

I have tabix working well locally but want to store my genomes on an S3 bucket (index files will likely stay local). I am having trouble getting this to work as advertised.

wget https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz

Resolving s3-eu-west-1.amazonaws.com... ...

Connecting to s3-eu-west-1.amazonaws.com|...|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 443976024 (423M) [application/x-gzip]

Saving to: 'WGC069899D-hg38-sorted.vcf.gz.1'

So no (permission) problems with accessing the file from this bucket.

But when I try tabix I get:

tabix https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz chr1:1000-2000

[E::hts_open_format] fail to open file 'https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz'

Could not read https://s3-eu-west-1.amazonaws.com/*/WGC069899D-hg38-sorted.vcf.gz

When I do the same tbix on the file locally, it works fine?

What am I missing?

Thanks.

tabix
• 4,855 views • link
updated by Santosh Anand Santosh Anand • written by andrewl • andrewl

What happens if you instead tabix s3://*/WGC069899D-hg38-sorted.vcf.gz chr1:1000-2000, where * is the bucket on S3? At least for bam/cram files that sort of thing works.

• 0 views • link
written by Devon Ryan Devon Ryan

1 answer

The problem might be the https: protocol. See this https://github.com/samtools/htslib/issues/139

TL; DR: Check that your htslib is current, and it has been installed with libcurl and libcrypto libraries. See requirements at https://github.com/samtools/htslib/blob/develop/INSTALL

HTSlib uses the following external libraries. Building requires both the library itself, and include files needed to compile code that uses functions from the library. Note that some Linux distributions put include files in a development ('-dev' or '-devel') package separate from the main library.

libz       (required)
libbz2     (required, unless configured with --disable-bz2)
liblzma    (required, unless configured with --disable-lzma)
libcurl    (optional, but strongly recommended)
libcrypto  (optional for Amazon S3 support; not needed on MacOS)
• 0 views • link
written by Santosh Anand Santosh Anand

^This. You need to build tabix with libcurl+openssl.

• 0 views • link
written by lh3 lh3

hello. Can you help me with the same issue. I am new to this environment of tabix and s3

• 0 views • link
written by shubhamnath5 • shubhamnath5

hello. Can you help me with the same issue. I am new to this environment of tabix and s3

• 0 views • link
written by shubhamnath5 • shubhamnath5

Log in to answer this question.

More posts like this