This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Downloading from MG-RAST

I am trying to download ~100 public amplicon sets from MG RAST I am using a slightly modified script from a previous entry here on BioStars (Downloading Data From Mg-Rast) that looks like this

cat names.csv | while read line; do curl http://api.metagenomics.anl.gov/1/download/"$line"?file=100.2 >$line.fna; done

where names.csv is a list of the numbers of the files in which I am interested.

This will write all the files for the entries names.csv like it is supposed to, but each of those files only contain an error message:

<html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>

I can download the files if I replace "$line" with the actual number in my browser or if I run the command

curl http://api.metagenomics.anl.gov/1/download/metagenomenumber?file=100.2 > metagenomenumber.fna

from my terminal window, but I would much prefer to not have to do each one manually.

Any ideas on why my shell script may be failing?

Thanks!

Christy

mg-rast
• 5,460 views • link
updated by Ram Ram • written by clm406 • clm406

Try replacing $line with ${line} maybe? Just an idea since I don't have access to your CSV file.

• 0 views • link
updated by Ram Ram • written by Dan D Dan D

1 answer

The URL of the resource to be downloaded should generally be completely enclosed in quotes to prevent any of its contents being interpreted by the shell as additional arguments or file patterns. In your case this means something like the following will work as you expect:

cat names.csv | while read line; do curl "http://api.metagenomics.anl.gov/1/download/$line?file=100.2" > "$line.fna"; done

As noted by Dan D, you might want to protect yourself further by making the variable boundaries explicit by using something like:

cat names.csv | while read line; do curl "http://api.metagenomics.anl.gov/1/download/${line}?file=100.2" > "${line}.fna"; done

Although this is unnecessary in this case.

• 0 views • link
updated by Ram Ram • written by hpmcwill hpmcwill

Is this question and answer about the Unix language for cURL? Is there a similar function for 'while' and 'do' for cURL in a windows command prompt? I found that 'type' is a similar command to 'cat'.

• 0 views • link
updated by Ram Ram • written by kbrannen • kbrannen

Log in to answer this question.

More posts like this