Hello,
Currently I'm trying to figure out how to download the latest patch of the hg38 assembly in AGP format from NCBI.
I have some oddly stringent requirements which makes this process difficult:
I've tried various combinations of wget recursive/accept-regex combinations but it seems almost ignored since I don't believe it fetches a proper html file since it refers to a FTP site. You can "glob" on FTP sites using wget, which fetches the ".listing" file and the glob matches the pattern in there but I cannot find a pattern that only matches the AGP files I'm interested in.
Any insight or best guesses would be greatly appreciated.
Thanks!
•
written
by
eroberts •
Use the following command to get just the agp files. Since the link below will always contain the latest files it should be reasonably foolproof:
wget -r -nd -H --reject "index.*" --accept "*chr*.agp.gz" --reject "*chrMT*" ftp://ftp.ncbi.nlm.nih.gov/genomes/Homo_sapiens/Assembled_chromosomes/agp/
Unfortunately, this retrieves the alt, MT, unplaced and unlocalized AGP files as well which doesn't suit my 3rd requirement. There is always the option of having this done in multiple commands by removing the unnecessary chromosomes afterwards but again I'm trying to keep this as short and simple as possible.
My mistake, yes I meant only MT file as well.
with parallel:
$ parallel --dry-run wget ftp://ftp.ncbi.nlm.nih.gov/genomes/Homo_sapiens/Assembled_chromosomes/agp/hs_ref_GRCh38.p12_{}.agp.gz ::: chr{1..22} chr{X..Y}
with wget:
$ wget ftp://ftp.ncbi.nlm.nih.gov/genomes/Homo_sapiens/Assembled_chromosomes/agp/hs_ref_GRCh38.p12_chr{X..Y}.agp.gz
$ wget ftp://ftp.ncbi.nlm.nih.gov/genomes/Homo_sapiens/Assembled_chromosomes/agp/hs_ref_GRCh38.p12_chr{1..22}.agp.gz
I've thought about using the parallel option, if I went that route I'd still have to worry about the patch number which may change.
Also with the 2nd wget command, it still runs into the issue of not being patch number agnostic. You can also put the bash sequences into a single "{}" like so: "{{1..22},{X,Y}}" which is what I had previously except for the issue with the patch number I just mentioned. I'm also not a big fan of using bash this way since it runs multiple wget commands, which in the case of the ftp access, means accessing and downloading the .listing file for each chromosome (which is not the worst just not preferred).
Log in to answer this question.
