samtools covert to bam, sort and index all gz.sam
The below bash script utilizes samtools in parellel to convert all gz.sam in a directory, sort, and index (I think). However, it seems to be processing a long time and I am not sure if it is indexed. Is there a better, more-efficient way? Without the .gz it is much faster. I am using samtools 1.9. Thank you :).
logfile=/path/to/fastq/process.log
dir=/path/to/fastq/
cd "$dir"
x=$(ls -dq *.sam* | wc -l)
echo "Starting conversion of" $x "sam files on" $(date) >> "$logfile"
ls *.sam | parallel "samtools view -b -S {} | samtools sort - {.}"
echo "conversion of" $x "sam files complete and coverted to sorted bam on" $(date) >> "$logfile"
1 answer
The question "not sure if it is indexed" is probably not relevant to sorting (indexing only happens after you have sorted bam files)
Other options for speedup
http://devblog.dnanexus.com/faster-bam-sorting-with-samtools-and-rocksdb/
https://www.basepairtech.com/blog/sorting-bam-files-samtools-vs-sambamba/
Log in to answer this question.
More posts like this
-
HLA alignment, approximation and frequencies
written by oumo •
-
Mapping paired end reads with ngm and samtools, using prefixes and suffixes for creating vcf event…
written by Human •
-
FASTQ to VCF pipeline question
written by Roy •
-
How to create custom docker image to download bam files from SRA
written by MAPK
-
Samtools Parse error at line 1
written by storm1907 •
-
[E::faidx_fetch_seq] error with Samtools mpileup?
written by mwerseb1 •
-
Error : viGEN : bowtie2 Alignment - No input read files were valid
written by venura
-
paired-end sequence alignment
written by evelyn
-
Qiime2 manifest file for multiple pair end read
written by Bioinfonext
-
Bash to match and store numerical prefix to update files
written by bioguy24
Less likely that there is going to be a faster way (you are already using
paralleland all cores you have access to locally?). Unless you access to a large cluster with hundreds of CPU's and a really high-performance file system where you can start all jobs at the same time.samtools sortaccept sam as input and can output in bam format. So there is no need to usesamtools viewbefore. But I don't know whether this is time consuming.Furthermore there is the
-@for using multiple threads. But again I don't know if you can save a lot of time with this. The bottleneck is the sorting itself.fin swimmer
From personal experience,
sambambaruns faster. After I made the switch, I haven't gone back to benchmarksamtoolswith the latest versions in the past couple of years, so the 2 tools might perform similarly now. Instead of ramping up all the available cores to run jobs simultaneously, providing more memory per sort will make it run a lot quicker. For instance, sort using 8 cores with 32GB of memory (4G/core) will very likely finish quicker than using 32 cores with 8GB of total memory. Also, if you have multiple storage options (i,e network-based vs instance-store in the cloud), set your temp directory to utilize the fastest storage.