bwa samtool giving empty bam files while piping

Question

[Goku]

Hi

I am getting empty bam files while using the following command. Any help is very much appreciated. Part of the output is also shown.

$ bwa mem -t 8 hg19.fa xxx_Q30_trim.fastq.gz | samtools view -bS - | samtools view -b -q 30 - | samtools sort - > xxx.bam

Thank you

[bam_header_read] EOF marker is absent. The input is probably truncated.

Usage: samtools sort [options] <in.bam> <out.prefix>

Options: -n sort by read name -f use <out.prefix> as full file name instead of prefix -o final output to stdout -l INT compression level, from 0 to 9 [-1] -@ INT number of sorting and compression threads [1] -m INT max memory per thread; suffix K/M/G recognized [768M]

[M::bwa_idx_load_from_disk] read 0 ALT contigs

[WouterDeCoster]

Are you using a reasonably recent samtools version? I think you can reduce your samtools view commands to just one.

Is there more output than what you showed? Please add everything.

[Goku]

Program: samtools (Tools for alignments in the SAM format) Version: 0.1.19-44428cd

I will try with the new version too. I had submitted 10 files, so the output is pretty huge.

But one thing i noticed it contained this line [main] Version: 0.7.17-r1188 [main] CMD: bwa mem -t 8 /home/rcf-proj2/hg1/software/hg19_index/hg19.fa WTDAC5_Q30_trim.fastq.gz [main] Real time: 7239.654 sec; CPU: 5991.808 sec Does this mean piping is not working. I will try other things that are suggested here.

[h.mon]

Maybe this means you are using nohup and forgot to tell us?

[Goku]

i don't know whats nohup is. But the - - thing worked for me that you suggested.

But I didn't get any output log file. generally i should have got one. I am using pbs job submission. I think this may be a different problem. Now I am not sure whether I should trust these bam files, they look fine. but i still should find out about those output log files....may be from system administrator.

Thanks,

Answer 1

[h.mon]

I don't know your SAMtools version, but as you use -S, I guess it is old. The following works for me with SAMtools 1.6:

bwa mem -t 8 hg19.fa xxx_Q30_trim.fastq.gz | \
    samtools view -b -q 30 -o - - | samtools sort -o xxx.bam -

I think the problem is you need two dashes for the samtools commands, if you want to redirect both standard input and standard output.

[Goku]

OK. So i got the new samtools Program: samtools (Tools for alignments in the SAM format) Version: 1.7 (using htslib 1.7) It looks very different from the old one.

also the - - thing worked. (I thought the stdout is suppose to be auto redirected without a dash. I just started using pipes and &)

Thanks a lot

[h.mon]

How are you using &? The & may work differently, depending on how you use it:

• stdout 2 stderr (1>&2)
• stderr 2 stdout (2>&1)
• stderr and stdout 2 file (&>file)

You probably piped stderr into stdout.

[WouterDeCoster]

Or just use the & at the end of the command to run it in the background (googling tells me it's not exactly the same as nohup though).

[Goku]

I use it at the end of line like this.

bwa mem -t 8 hg19.fa xxx1_Q30_trim.fastq.gz | samtools view -b -q 30 -o - - | samtools sort - -o xxx0.bam &
bwa mem -t 8 hg19.fa xxx5_Q30_trim.fastq.gz | samtools view -b -q 30 -o - - | samtools sort - -o xxx5.bam

[WouterDeCoster]

The & is nohup and likely the source of your problem