Bowtie2 through tophat2 issue : Error reading RefRecord 'first' flag

Question

[Rox]

Hi everyone !

I have some trouble with using bowtie2 through tophat2.

I'm trying to align a bunch of reads against my assembly :

Loutre:~$ tophat2 --num-threads 25 --max-intron-length 50000 -o \
> /media/loutre/SUZUKII/annotation/tophat_out \
> /media/loutre/SUZUKII/annotation/indexes/cleaned_canu3_suzukii \
> '/media/loutre/SUZUKII/annotation/evidences/suz_ant.R1.fastq.gz'\
> ,'/media/loutre/SUZUKII/annotation/evidences/suz_ovi.R1.fastq.gz'\
> ,'/media/loutre/SUZUKII/annotation/evidences/suz_pro.R1.fastq.gz'\
> ,'/media/loutre/SUZUKII/annotation/evidences/suz_tar.R1.fastq.gz'\
>  '/media/loutre/SUZUKII/annotation/evidences/suz_ant.R2.fastq.gz'\
> ,'/media/loutre/SUZUKII/annotation/evidences/suz_ovi.R2.fastq.gz'\
> ,'/media/loutre/SUZUKII/annotation/evidences/suz_pro.R2.fastq.gz'\
> ,'/media/loutre/SUZUKII/annotation/evidences/suz_tar.R2.fastq.gz'

This command seem to work at first, but after approximatively 2hours, it throws an bowtie2 error message :

[2016-08-16 13:31:56] Beginning TopHat run (v2.0.9)

[2016-08-16 13:31:56] Checking for Bowtie
          Bowtie version:    2.1.0.0
[2016-08-16 13:31:56] Checking for Samtools
        Samtools version:    0.1.19.0
[2016-08-16 13:31:56] Checking for Bowtie index files (genome)..
[2016-08-16 13:31:56] Checking for reference FASTA file
[2016-08-16 13:31:56] Generating SAM header for /media/loutre/SUZUKII/annotation/indexes/cleaned_canu3_suzukii
    format:      fastq
    quality scale:   phred33 (default)
[2016-08-16 13:31:56] Preparing reads
     left reads: min. length=100, max. length=100, 180811539 kept reads (47437 discarded)
    right reads: min. length=100, max. length=100, 180039152 kept reads (819824 discarded)
[2016-08-16 15:05:07] Mapping left_kept_reads to genome cleaned_canu3_suzukii with Bowtie2 
    [FAILED]
Error running bowtie:
Error reading RefRecord 'first' flag
Error: Encountered internal Bowtie 2 exception (#1)
Command: /usr/bin/bowtie2-align -q -k 20 -D 15 -R 2 -N 0 -L 20 -i S,1,1.25 --gbar 4 --mp 6,2 --np 1 --rdg 5,3 --rfg 5,3 --score-min C,-14,0 -p 25 --sam-no-hd -x /media/loutre/SUZUKII/annotation/indexes/cleaned_canu3_suzukii -

Am I doing something wrong ?

Thanks for helping,

Cheers,

Roxane

[Rox]

And this error also happen when I use bowtie2 only, without calling tophat2, here my command :

bowtie2 -p 8 -x /media/DATAPART1/Documents/suzukii_assembly/annotation/indexes/cleaned_canu3_suzukii
-1 /media/DATAPART1/Documents/suzukii_assembly/annotation/evidences/suz_ant.R1.fastq.gz
-2 /media/DATAPART1/Documents/suzukii_assembly/annotation/evidences/suz_ant.R2.fastq.gz

Here the same error I get :

(...)
@SQ SN:tig00002212  LN:38381
@PG ID:bowtie2  PN:bowtie2  VN:2.1.0
Error reading RefRecord 'first' flag
Error: Encountered internal Bowtie 2 exception (#1)

[Rox]

An other hint, when I tried with an other index file (canu3_suz), it worked. Some clarifications : I manually cleaned canu3_suz (I throw away some contigs that was redundant) to obtain cleaned_canu3_suzukii. Bowtie2 work well on canu3_suz, but not on the cleaned_canu3_suzukii... Could that come from the way I am deleting contigs ?

I'm determining which contigs to throw with an home made bash script. At the end, I have a list of all contigs to delete, and a python script remove theses contigs :

python python/Remove_seqs_from_fasta_using_name.py $FASTA_FILE $DISCARDED_STRING > $CLEANED_OUTPUT

Here the python script that remove the contig for me :

# Input             - fasta file
#                   - name of the fasta seq to select (NO ">"). Only the field after the ">" will be considered. The text after a space will NOT be considered.
# Output            - directly the cleaned fasta file.

import os
import sys

#Where the sequences to blast are put
file = sys.argv[1]
lnames=list() # list of seq names to remove
for i in range(2,len(sys.argv)):
    lnames.append(sys.argv[i])

try:
    infile=open(file, 'r')
except IOError, e:
    print "Unknown file: ",file
    sys.exit()

OK=True
for line in infile:
    if line[0] == ">": # we enter a new sequence.
        name=line.split()[0].split(">")[1]
        if name in lnames: # is it a bad one that needs to be removed
            OK=False
        else: # We have entered a good sequence.
            OK=True
            print line.strip()
    else:# we are not within a sequence header. 
        if OK: #within an OK sequence. To keep
            print line.strip()
infile.close()

Do you see something wrong ?

[microfuge]

Could you please try running bowtie2-inspect -s -v <refindex> ? . Maybe something pops up.

[Rox]

Here what I tried, everything seem to look fine in bowtie2_inspect.log, in detect everything as contig

Loutre:~$ bowtie2-inspect -s -v /media/loutre/SUZUKII/annotation/indexes/cleaned_canu3_suzukii >> bowtie2_inspect.log
  About to open input files: 13:07:46
Opening "/media/loutre/SUZUKII/annotation/indexes/cleaned_canu3_suzukii.1.bt2"
  Finished opening input files: 13:07:46
  Reading header: 13:07:46
Headers:
    len: 281142694
    bwtLen: 281142695
    sz: 70285674
    bwtSz: 70285674
    lineRate: 6
    offRate: 4
    offMask: 0xfffffff0
    ftabChars: 10
    eftabLen: 20
    eftabSz: 80
    ftabLen: 1048577
    ftabSz: 4194308
    offsLen: 17571419
    offsSz: 70285676
    lineSz: 64
    sideSz: 64
    sideBwtSz: 48
    sideBwtLen: 192
    numSides: 1464285
    numLines: 1464285
    ebwtTotLen: 93714240
    ebwtTotSz: 93714240
    color: 0
    reverse: 0
Reading plen (1373): 13:07:46

And the output is exactly the same (except lengths) for the non-cleaned file which work with bowtie2

[GenoMax]

Why are you using versions of TopHat/Bowtie2/Samtools that are several years old? With TopHat suite it is always best to upgrade to the latest since many of these types of errors have been addressed over the years.

[Rox]

I've just downloaded them using apt-get install a few weeks ago, so for me I was downloading the latest version, and not a several years ago version.

[GenoMax]

The versions based on the log above are several years old. apt-get is not the foolproof way to get the latest versions. I suggest that you download the executables directly from TopHat/Bowtie2 sites. Samtools you will need to compile yourself.

TopHat (https://ccb.jhu.edu/software/tophat/downloads/tophat-2.1.1.Linux_x86_64.tar.gz )
Bowtie2 (https://sourceforge.net/projects/bowtie-bio/files/bowtie2/2.2.9/bowtie2-2.2.9-linux-x86_64.zip/download )
Samtools (http://www.htslib.org/download/ )

[Rox]

Thank you, it actually worked when I upgraded everything, thanks for the suggestion !