This is a test version of Biostars. For the public version, visit https://www.biostars.org.
download refseq of thousand of assembly file from NCBI

I want to download many bacteria fasta files with the .fna.gz extension from NCBI i have tried the commands below but none of them is working as it should. I do get the directory not the fasta files. Can anyone let me know what i should change to get the the ref seq of fasta files?

wget -b -r --no-parent -A 'GCF_*_genomic.fna.gz'  ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/bacteria/ 

wget -b -r --no-parent accept-regex=*/latest_assembly_versions/*/*_genomic.fna.gz ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/bacteria
bash wget refseq fasta
• 6,920 views • link
written by Shelle • Shelle

Modify @5heikki's solution in how to download all the complete genomes for mycobacteria from NCBI?. It refers to just Mycobaterial genomes but you can remove that restriction.

$ wget ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/assembly_summary_refseq.txt
$ cat assembly_summary_refseq.txt \
    | awk 'BEGIN{FS="\t"}{print $20}' \
    | awk 'BEGIN{OFS=FS="/"}{print $0,$NF"_genomic.fna.gz"}' \
    > urls.txt

Limit to list you have from bacterial directory.

• 1 views • link
written by GenoMax GenoMax

Thanks for your answer but what i want is the fasta file from this website : ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/bacteria/ link below can be one of the files that i am interested in:

ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/bacteria/Abditibacterium_utsteinense/latest_assembly_versions/GCF_002973605.1_ASM297360v1/GCF_002973605.1_ASM297360v1_genomic.fna.gz

I used the command you mentioned in your post but it doesn't give me right files!

• 1 views • link
written by Shelle • Shelle

That is not the final command. As I said you will need to limit the URL's produced if you need just bacterial data.

You may be better off using the program @joe included below. There you can just say ncbi-genome-download bacteria to get all bacterial genomes.

• 1 views • link
written by GenoMax GenoMax

Thanks. I am using this. It gives me bunch of different directories. When i actually go into each directory, there is a file with content like below:

d3d4a4c01a15dee5a054b38a3178bf12  ./GCF_000007725.1_ASM772v1_assembly_report.txt
c132f1a3ba2b00383f2a1d92e4460e2b  ./GCF_000007725.1_ASM772v1_assembly_stats.txt
7a2f6dc85caefaf326362077f72bb1ad  ./GCF_000007725.1_ASM772v1_cds_from_genomic.fna.gz
7e65c3da25f5a35d8a7860d6c478bf67  ./GCF_000007725.1_ASM772v1_feature_count.txt.gz
2d82d4315ca7a2004a3b03bc55aa42af  ./GCF_000007725.1_ASM772v1_feature_table.txt.gz
576cc643ef00d289009c95518f3792f5  ./GCF_000007725.1_ASM772v1_genomic.fna.gz
5a491b9ae2550dd9b6379e4f9054c4a2  ./GCF_000007725.1_ASM772v1_genomic.gbff.gz
25b139d63e6cd46484ac27daa8532b79  ./GCF_000007725.1_ASM772v1_genomic.gff.gz
10e61215025d12b872b28847e4a389fa  ./GCF_000007725.1_ASM772v1_protein.faa.gz
88b5db0e6f27fd5455e76d2b9180a67b  ./GCF_000007725.1_ASM772v1_protein.gpff.gz
64ae08d0ceff2696234aded52fdf8955  ./GCF_000007725.1_ASM772v1_rna_from_genomic.fna.gz
02c785fd2336a0cc3fd20687d3053460  ./GCF_000007725.1_ASM772v1_translated_cds.faa.gz
313d29e74f85d37ae6d701f606f1acac  ./annotation_hashes.txt

I am only interested in "./GCF_000007725.1_ASM772v1_genomic.fna.gz". Does anyone know how i can extract this and work with it separately?

• 1 views • link
written by Shelle • Shelle

You could use something like find . -name "*.fna.gz" and move those files to a new location and then delete the rest of the files if you don't want to keep them.

• 1 views • link
written by GenoMax GenoMax

The command you mentioned doesn't work unfortunately. I used this command "grep -w "GCF_000007365.1_ASM736v1_genomic.fna.gz" MD5SUMS.txt >> newfile/new.txt" to separate the fasta file but as a result the content of new.txt file would be something like below:

576cc643ef00d289009c95518f3792f5  ./GCF_000007725.1_ASM772v1_genomic.fna.gz

which isn't useful again as i want to work with this fasta file later on like decompress it...

• 1 views • link
written by Shelle • Shelle

For genomax's command to work you need to be working in a directory at the top of the tree. find is absolutely one of the best ways to do what you want, so you'll need to give us more info about what didn't work.

• 1 views • link
written by Joe Joe

It worked thanks for your comment about on the top of tree! But how i can decompress them when they are in the format like below:

./bacteria/bacteriagz/GCF_000008885.1_ASM888v1_genomic.fna.gz
./bacteria/bacteriagz/GCF_000009305.1_ASM930v1_genomic.fna.gz
./bacteria/bacteriagz/GCF_000019705.1_ASM1970v1_genomic.fna.gz
./bacteria/bacteriagz/GCF_000024505.1_ASM2450v1_genomic.fna.gz
./bacteria/bacteriagz/GCF_000147695.2_ASM14769v3_genomic.fna.gz
./bacteria/bacteriagz/GCF_000156275.1_ASM15627v1_genomic.fna.gz
./bacteria/bacteriagz/GCF_000285255.1_ASM28525v1_genomic.fna.gz
./bacteria/bacteriagz/GCF_000287295.1_ASM28729v1_genomic.fna.gz
./bacteria/bacteriagz/GCF_000831225.1_ASM83122v1_genomic.fna.gz
./bacteria/bacteriagz/GCF_001700895.1_ASM170089v1_genomic.fna.gz
./bacteria/bacteriagz/GCF_001705605.1_ASM170560v1_genomic.fna.gz
./bacteria/bacteriagz/GCF_002083165.2_ASM208316v2_genomic.fna.gz
./bacteria/bacteriagz/GCF_002257505.1_ASM225750v1_genomic.fna.gz
./bacteria/bacteriagz/GCF_002849875.1_ASM284987v1_genomic.fna.gz
./bacteria/bacteriagz/GCF_002855775.1_ASM285577v1_genomic.fna.gz
./bacteria/bacteriagz/GCF_003019755.1_ASM301975v1_genomic.fna.gz
./bacteria/bacteriagz/GCF_003019785.1_ASM301978v1_genomic.fna.gz
./bacteria/bacteriagz/GCF_003034925.1_ASM303492v1_genomic.fna.gz
./bacteria/bacteriagz/GCF_003043915.1_ASM304391v1_genomic.fna.gz
• 1 views • link
updated by Joe Joe • written by Shelle • Shelle

Are you familiar with genomax's use of the . syntax with find? It's shorthand for my current directory.

More fully that command would look like:

find /path/to/search/from -name "some_string"
• 1 views • link
written by Joe Joe

It worked thanks. But what i don't know is how i can use the gz files which is outputted from the find command. I need to decompress them as i need to work with the unzip format of fasta file later on! but i don't know how i can do it when it is in this format!

• 0 views • link
written by Shelle • Shelle

Did you try Googling unzip gz files? We cannot help you if you don't invest some effort by yourself first.

• 1 views • link
written by Ram Ram

I actually did this! But I am doing something wrong. I save all the files which is outputted from 'find' command to a text file named as new.txt and i used command in below

for i in new.txt;do
   gunzip $i
done

but gives me -bash: ./file: bash.sh: bad interpreter: No such file or directory

• 1 views • link
written by Shelle • Shelle

You can't do that. Spend some time getting familiar with bash if you're going to be analysing thousands of sequences. It will pay off in the long run.

What you're trying to do would need to look more like:

#!/bin/bash
while read file ; do
   gunzip $file
done < new.txt
• 1 views • link
written by Joe Joe

or cat new.txt | xargs -I file_i gunzip file_i

• 1 views • link
written by Ram Ram

I already commented on this below. This is super easy to google.

• 1 views • link
written by Joe Joe

Hi jrj.healey, I noticed this command 'find . -name ".fna.gz" ' is not working at all. Even if i changed '.' to my current path where all the downloaded bacteria directories are, no result when running the command. Even if i was at the top of directory, it gives me nothing! Earlier that i commented it worked, it was my mistake. I had some other fasta bacteria in a different directory and when i used 'find . -name ".fna.gz" ' those fasta files showed up. I deleted those to confirm if this command is working or not. It turned out when i downloaded the bacteria directories with this ncbi command, the find command in any of the format i have used is not working. Any idea to solve this issue?

• 1 views • link
written by Shelle • Shelle

1 answer

You should be able to use ncbi-genome-download for this I think.

https://github.com/kblin/ncbi-genome-download

• 1 views • link
written by Joe Joe

Log in to answer this question.

More posts like this