This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Remotely Access Bigwig File With Python

I use bx-python for reading local bigWig files, but I don't believe it works on remote files like bigWigSummary from kentsrc does, e.g., 

bigWigSummary <http://file.bigWig> chr1 1 100000 10

I suppose one option would be to loosely wrap bigWigSummary, but does anyone know of an existing way of accessing bigWigs remotely with Python?

python bigwig
• 4,657 views • link
updated by Michael Schubert Michael Schubert • written by Ryan Dale Ryan Dale

I know this is probably not an option for you: perl bindings do support remote files.

• 0 views • link
written by lh3 lh3

Yeah, the goal is to integrate with a lot of other Python code. But thanks for this -- time to read up and see how they did it.

• 0 views • link
written by Ryan Dale Ryan Dale

1 answer

What you're asking is not exactly possible.

You pass the remote URL to bigWigSummary as command line argument. This means:

  • the program needs to be able to interpret the remote URL as such and
  • even if it does, programs on your computer still can't analyse remote data without at least temporarily fetching them

Possible solutions are:

  • download the file, run your program, and delete the file
  • have bigWigSummary on the remote machine where the data is and control it there (SSH)
• 0 views • link
written by Michael Schubert Michael Schubert

The whole point of bigwig is to only download a tiny fraction of the remote file in the region we are interested in, as long as the ftp/http server allows to resume broken transfer. I do not know the bigwig APIs. For BAM/tabix, it is the C APIs that transparently connect to the remote server given a file name prefixed with "http://" or "ftp://". Perl/Python bindings do not need to worry about parsing URL or establishing the connection.

• 0 views • link
written by lh3 lh3

Interesting point, but then the program would have to guess the byte ranges, download a chunk of the file, see if the required data is in there, and search like this with multiple (possibly hundreds of) requests?- unless there is some specific index that it could fetch before.

• 0 views • link
written by Michael Schubert Michael Schubert

For bam/tabix, users need to download a ~10MB index file to local disk. Once they have this file, they can access most of alignments/records with only one http/ftp request (<1.1 requests in average). The index is carefully tuned for fewer requests. BigBed is more advanced. The index is integrated in data. You do not need to download the entire index. A tradeoff though is that it requires more requests (I think about 8 in average) to jump in the index section of the file. BigWig should be similar.

• 0 views • link
written by lh3 lh3

Log in to answer this question.

More posts like this