Thank you for using NGSlib. This package is developed in my spare time so I am sorry that I don't have enough time to maintain it and update regularly.

The previous answers were correct about when to close the BigWigFile. But this is not the reason for your problem. The file handle doesn't take a lot of memory, and the memory will be released when the bw object is destroyed when it is out of its scope. See BigWigFile.__exit__().

The answers provided by joe.cornish826 et al. were correct. The problem you mentioned is not a problem of the code. It's the memory usage strategy of Python. When you load a whole chrom from a huge file into memory, it is considered as a big memory block and is reversed for Python to use it later. The memory will be released when the python script is finished. You may read related topics on the website. This is a common problem for advanced language. Most of the time, you don't need to care about the memory releasing since Python will do it for you. But you may have problem in some extreme cases.

The BigWigFile is actually a wrapper of Kent's UCSC codes. It is used for fast retrieval of items or depth in a short genomic region, not for the whole chromosome (use bash to do whole genome). When you want to go through the whole file, there are many other strategies. The recommended strategy are:

1. If you really want to fetch the the whole genome, you may use bash.

BigWigToWig test.bw | yourscript_to_catch_standard_output_as_input.py

or convert it to wig file and manipulate on the wig file.

2. Put the fetching into a single script. When it is finished, Python will release the memory. Do not do multiple fetching in the same script.

3. Split your chromosome into bins (10K for example), fetch a bin at each time.

4. Follow codes in external/KentLib/wWigIO/wWigIO.c, find a way to call it in C instead of Python.

By the way, if you are going to get the scores at each position instead of all the items, you may use the pileup function. It returns a numpy array with length = end-start. Remember to split the chromosome into bins, otherwise the problem will still be there.

Hope this helps.

Instead of using Python, you could use BEDOPS bedmap and wig2bed with bash process substitution to get the --mean signal over some ad-hoc interval:

$ bigWigToWig signal.bigWig signal.wig
$ echo -e 'chr1\t1234\t3456' | bedmap --echo --mean - <(wig2bed < signal.wig) > answer.bed

In this case, answer.bed contains mean signal value over the ad-hoc interval chr1\t1234\t3456.

If you have a sorted BED file of intervals (intervals.bed) over which you want mean signal:

$ bigWigToWig signal.bigWig signal.wig
$ bedmap --echo --mean intervals.bed <(wig2bed < signal.wig) > answer.bed

In this case, answer.bed contains mean signal values over each provided interval.

One more option, perhaps. Command-line options are fast, concise and not dependent on the vagaries of Python garbage collection.

Well, I have never used (or heard of) ngslib, and its documentation is just amazing, but you are calling bw.close() after return, meaning it will never get executed -- maybe that's the problem.

EDIT: In theory, the correct pythonic way to write your code is:

bw = BigWigFile(bigWig)

try:
    # stuff goes here
    return mean_score
finally:
    bw.close()

This way the bw.close() command will be executed both whenever there is an exception raised in the try block (and thus cleaning up), and after the return command is executed (which is also nice for you).

If the library developers implemented the context manager support (and I doubt it, but nevertheless including this for completeness) even better way to write this would be:

with BigWigFile(bigWig) as bw:
     # do work   
     return mean_score
 # bw.close() should be called automatically when bw is out of scope.

Python doesn't release the memory it has allocated after whatever was located in that memory has been deleted. This isn't exactly true, but the details aren't as important here.

http://effbot.org/pyfaq/why-doesnt-python-release-the-memory-when-i-delete-a-large-object.htm

However, this memory is still available within Python. Basically by holding onto already allocated pieces of memory, the interpreter can just fill them back up again rather than immediately releasing them to the OS and having to allocate them. So it isn't like that memory is still consumed and can't be used again. The idea is to prevent fragmenting the memory and remove the overhead of calls to malloc and free the interpreter would have to make.

http://www.evanjones.ca/memoryallocator/

Generally when you get a segfault with Python it is because you're out of memory.

Your script shows you opening and closing the file each time your function is called, I would avoid this. It would be better to pass the opened file or use the 'with open...' syntax. You could also pass a list of intervals to your function. This doesn't solve the memory issue however.

Without any documentation for the function BigWigFile, I can't really tell what it is doing. I'm guessing that it is reading the whole file into memory. Unless you can change it to use a generator, you might be in trouble.

You could try splitting the file up and then for each part of the file you parse the genomic intervals associated with it.