Reading vcf file using python gives UnicodeDecodeError

python version Python 3.5.4

Trying to process vcf file as shown in the code below:

 with open(myfile, 'r') as datain:
    for myline in datain:
        if myline.startswith("#"):
            pass
        else:
            do something

In some line (after processing couple of hundreds of lines) it raises this error:

UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9d in position 8077: invalid start byte

Any idea how to fix this error?

python vcf

• 8,087 views • link

written 7.5 years ago by Medhat

Did you try this? - https://stackoverflow.com/questions/22216076/unicodedecodeerror-utf8-codec-cant-decode-byte-0xa5-in-position-0-invalid-s

• 1 views • link

written 7.5 years ago by Kevin Blighe

This se answer is related, but dealing with a python 2 encoding problem. python 3 behaviour is fairly different in this area.

• 1 views • link

written 7.5 years ago by e.benn

As you mentioned it is already decoded.

• 1 views • link

written 7.5 years ago by Medhat

@ Kevin Blighe If I used the code like: myline = myline.encode('utf-8').strip() It will give this error:

TypeError: startswith first arg must be bytes or a tuple of bytes, not str Then I could use str() but this will not help in fixing the main issue.

• 1 views • link

written 7.5 years ago by Medhat

Are you using an European locale setting by any chance? What does locale produce? ~~You may need to force the locale to be UTF-8~~. See this: https://unix.stackexchange.com/questions/87745/what-does-lc-all-c-do

• 1 views • link

written 7.5 years ago by GenoMax

The error message shows that python is using utf8 to decode. The question is what codec of the input

• 1 views • link

written 7.5 years ago by e.benn

2 answers

Python 3.0+ decodes files by default.

https://docs.python.org/3/library/functions.html#open

"the contents of the file are returned as str, the bytes having been first decoded using a platform-dependent encoding or using the specified encoding if given."

Apparently your file is not encoded in utf-8. My guesses would be that it is gzipped, or encoded in some other encoding. What happens if you file myfile.vcf ? Does it report gzip encoded data? Does it report the codec?

Try:

# read as binary (~ascii) data (like python 2)
with open(myfile, 'rb') as datain:

# specify the actual encoding
with open(myfile, 'r', encoding="ascii") as datain:

# the vcf is gzipped
with gzip.open(filename) as datain:

• 1 views • link

written 7.5 years ago by e.benn

Actually, it is not gzipped file. If I read as binary 'rb' I need to convert it again to string.

• 1 views • link

written 7.5 years ago by Medhat

The vcf spec just states 'text' encoding. https://samtools.github.io/hts-specs/VCFv4.2.pdf Most likely you have ascii. Specify ascii as the encoding parameter to open. There is some chance you don't have ascii if the file has been moved between platforms. What does the file program say?

• 1 views • link

written 7.5 years ago by e.benn

The file is data also the loop does not fail in the first line it reads couple of hundreds of line then raise that error.

• 1 views • link

written 7.5 years ago by Medhat

when using: with open(myfile, 'r', encoding="ascii") as datain:

It changes the error to : UnicodeDecodeError: 'ascii' codec can't decode byte 0x9d in position 8077: ordinal not in range(128)

• 1 views • link

written 7.5 years ago by Medhat

So I manage to work around , but did not fix the issue.

The original file was a result of using bcftools merge A.vcf B.vcf C.vcf > someresult.vcf and that gave me the error above.

But when using vcf-merge A.vcf B.vcf C.vcf > someresult.vcf The issue was fixed.

This does not clearly answer my question, But it fixed the issue.

• 1 views • link

written 7.5 years ago by Medhat

Always specify output format with bcftools, it screws you over otherwise.

• 1 views • link

written 7.5 years ago by Ram

I tryied your suggestion using:

-O z -o result.vcf.gz

But still have (result, consumed) = self._buffer_decode(data, self.errors, final) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x9d in position 8116: invalid start byte

• 1 views • link

written 7.5 years ago by Medhat

Does reading the individual files work OK?

• 1 views • link

written 7.5 years ago by Ram

Yes I will just stay with file from vcftools

• 1 views • link

written 7.5 years ago by Medhat

When you specify to bcftools that the output is to be gzipped, you get the same error. This again suggests to me that the original problem was a gzipped vcf file. What is the result of running file myfile.vcf

• 1 views • link

written 7.5 years ago by e.benn

The old file was data as I mentioned in a comment on your answer. C: Reading vcf file using python gives UnicodeDecodeError

• 1 views • link

written 7.5 years ago by Medhat

Apologies, I missed that. On my system a .vcf file is reported as : Variant Call Format (VCF) version 4.1, ASCII text A .vcf.gz files is reported as: gzip compressed data, extra field. It sounds like the bcftools command may be making a corrupt file, maybe a bug report would be useful. Did you check the bcftools command for an error code? What happens if you bcftools view the merged file? Which bcftools version?

• 1 views • link

written 7.5 years ago by e.benn

I do not have access now But what you are suggesting is really interesting, I will try and keep you updates (when using bcftools there were no errors)

• 1 views • link

written 7.5 years ago by Medhat

Hopefully you can get to the bottom of the issue. When I encounter a data problem like this I get scared - if such a well used program as bcftools has an issue with the input it could mean the data is corrupt - maybe the alternative tool vcf-merge is failing in a different way that seems to work, but is actually giving you garbage data.

• 1 views • link

written 7.5 years ago by e.benn

There are always edge cases that make even well-tested tools like bcftools fail. An example is using bcftools norm on an indel-heavy VCF - I ran into a scenario where it made an entry with identical chr, pos, ref and alt entries, with just the DP/DQ/something-of-that-sort different.

• 1 views • link

written 7.5 years ago by Ram