This is a test version of Biostars. For the public version, visit https://www.biostars.org.
how to download all the complete genomes for mycobacteria from NCBI?

How to download all the complete genomes for mycobacteria from NCBI?

I tried downloading the complete genomes from the NCBI site

ftp://ftp.ncbi.nlm.nih.gov/genomes/GENOME_REPORTS/

But couldn't get the exact fasta files with respective mycobacteria. And https://www.ncbi.nlm.nih.gov/genome/?term=mycobacteria gave me 421 hits

genome ncbi sequence
• 5,425 views • link
updated by Debut • Debut • written by Paul Paul

1 answer 

#Get GenBank assembly summary file
wget ftp://ftp.ncbi.nlm.nih.gov/genomes/genbank/assembly_summary_genbank.txt

#Get all lines that have "Mycobacter", if 12th field is "Complete Genome", print the 20th field (url to file).
#But the actual filename ends _genomic.fna.gz so include that too..
grep Mycobacter assembly_summary_genbank.txt \
    | awk 'BEGIN{FS="\t"}{if($12=="Complete Genome"){print $20}}' \
    | awk 'BEGIN{OFS=FS="/"}{print $0,$NF"_genomic.fna.gz"}' \
    > urls.txt

#Now you can go through your urls file
IFS=$'\n'; for NEXT in $(cat urls.txt); do wget "$NEXT"; done
• 0 views • link
written by 5heikki 5heikki

Thanks.. This worked

• 0 views • link
written by Paul Paul

I tried your method but I have an empty urls.txt file. has the format changed please?

• 0 views • link
written by Debut • Debut

It hasn't changed. I just tried the above and see 2,481 Mycobacter genomes with the status "Complete Genome"..

• 0 views • link
written by 5heikki 5heikki

OKAY, THANK YOU FOR YOUR ANSWER.

• 0 views • link
written by Debut • Debut 
$ grep klebsiella assembly_summary_genbank.txt | awk 'BEGIN{FS="\t"}{if($12=="Complete Genome"){print $20}}' | wc -l
0
$ grep Klebsiella assembly_summary_genbank.txt | awk 'BEGIN{FS="\t"}{if($12=="Complete Genome"){print $20}}' | wc -l
1523
• 0 views • link
written by 5heikki 5heikki

please, is it possible to put all the output sequences in one file (file with several FASTA files) ?

• 0 views • link
written by Debut • Debut 
$ ls
file1.fna  file2.fna
$ cat file1.fna
>seq1
aaaaaaaaaa
$ cat file2.fna
>seq2
gggggg
$ cat file1.fna file2.fna > file3.fna
$ cat file3.fna
>seq1
aaaaaaaaaa
>seq2
gggggg
• 0 views • link
written by 5heikki 5heikki

thank you very much for your answer. but i have 10668 outputs it doesn't have a command to add for example after "IFS=$'\n'; for NEXT in $(cat urls.txt); do wget "$NEXT"; done" i tried IFS=$'\n'; for NEXT in $(cat urls.txt); do wget "$NEXT"; done >doc.txt" it didn't work

• 0 views • link
written by Debut • Debut

The output files all end in ".gz", right?

So zcat *.gz > all.fna

zcat instead of cat because they're gz archieves

• 0 views • link
written by 5heikki 5heikki

Log in to answer this question.

More posts like this