This is a test version of Biostars. For the public version, visit https://www.biostars.org.
samtools sort error

Hello All,

I am running bwa-mem with compressed paired-end input files using an array. I am trying to make sorted bam files as well. I am using this array:

INPUT=$(ls -1 input/*_1.fq.gz | sed -n ${RUN}p)
SAMPLE=$(basename "$INPUT" | sed 's/_1.fq.gz//')

bwa mem GENOME ${SAMPLE}_1.fq.gz ${SAMPLE}_2.fq.gz | samtools sort -o ${SAMPLE}_sorted.bam

However, at some point it stops running with an error:

[E::hts_open_format] fail to open file 'sorted.bam.tmp.0000.bam'
[bam_sort_core] failed to create temporary file "sorted.bam.tmp.0000.bam": File exists

I used the same array for uncompressed files before and there was no error. I am trying to save some space by using compressed files.

I will appreciate any input. Thanks!

snp
• 3,477 views • link
updated by WouterDeCoster WouterDeCoster • written by evelyn evelyn

basename takes a parameter to trim, you don't need the sed. Even without that, you could use ${INPUT/%_1.fq.gz/}

INPUT=ABCD_1.fq.gz
basename $INPUT _1.fq.gz
ABCD #Output
echo ${INPUT/%_1.fq.gz/}
ABCD #Output
• 0 views • link
written by Ram Ram

You ran this command, or similar, before in this directory. These temporary files need to be removed first.

• 0 views • link
written by WouterDeCoster WouterDeCoster

Thank you! I tried this. I removed the temporary files from previous runs. I also tried running it after making a new directory. But it always gives the same error.

• 0 views • link
written by evelyn evelyn

1 answer

You're overwriting your output file. Use -o ${SAMPLE}_sorted.bam instead of just -o sorted.bam

• 0 views • link
written by Ram Ram

This is my mistake to not mention it in the question correctly. I am running as you suggested but I didn't write it. I have edited my question. Thank you!

• 0 views • link
written by evelyn evelyn

If the error persists you are still overwriting things. Check if ${SAMPLE} indeed contains information or if it is constantly empty because you did not correctly feed in information.

• 0 views • link
written by ATpoint ATpoint

echo $SAMPLE gives the sample names in the directory. It does not show as it is empty.

• 0 views • link
written by evelyn evelyn

I have a feeling that your $SAMPLE has all the samples in a single string, breaking your bwa mem command. Can you try:

echo "bwa mem GENOME ${SAMPLE}_1.fq.gz ${SAMPLE}_2.fq.gz"

instead of the bwa mem | samtools sort line?

• 0 views • link
written by Ram Ram

I ran it and shows one sample out of the four in that directory.

bwa mem GENOME sample/USD1_1.fq.gz sample/USD1_2.fq.gz
• 0 views • link
written by evelyn evelyn

The problem is with how you pick your $INPUT. Try a loop:

for f in input/*_1.fq.gz
do
INPUT=$(sed -n ${RUN}p $f)
SAMPLE=${INPUT/%_1.fq.gz/}
echo "bwa mem GENOME ${SAMPLE}_1.fq.gz ${SAMPLE}_2.fq.gz | samtools sort -o ${SAMPLE}_sorted.bam"
done

If the commands echoed look fine, remove the echo and the double quotes and you'll be all set.

• 0 views • link
written by Ram Ram

I am not sure if I would switch to a loop especially when this array worked for my similar jobs earlier. I am still trying to find out why I am getting an error regarding .tmp files.

• 0 views • link
written by evelyn evelyn

If the loop works, you'll know the problem lies with the array structure and not the code being executed. It helps you in the process of elimination.

• 0 views • link
written by Ram Ram

Log in to answer this question.

More posts like this