Bash loop using Edirect (esearch|efetch)

Question

[fhsantanna]

I have a text file containing in each line an access number (file 16S_list). I tried to download sequences from genbank using this list through the following script:

#!/bin/bash
while read line; do
esearch -db nucleotide -query $line | efetch -db nucleotide -format fasta 
done < 16S_list

However, it did not worked, just the first sequence is downloaded. After I tried the following script, and this time, it worked:

#!/bin/bash
while read line; do
esearch -db nucleotide -query $line < /dev/null | efetch -db nucleotide -format fasta
done

I did not understand why the first one is not working. Could somebody explain why the first script failed?

[Pierre Lindenbaum]

do you have any empty line in 16S_list ? try to quote $line to "$line"

I don't get why the second script would work by reading /dev/null

[5heikki]

Does the first one work if you set IFS to newline? IFS=$'\n'? Are your lines delimited by \n? I recall having problems with this too. AFAIR the esearch command consumed the entire file (stdin), hence I got output only for the first line. I think I ended up using IFS=$'\n'; for next in $(cat file); do ..; done

[fhsantanna]

It worked. Here is the complete code:

IFS=$'\n'; for next in $(cat 16S_list); do esearch -db nucleotide -query $next | efetch -db nucleotide -format fasta; done

However, I still do not understand why my first script did not worked.

[5heikki]

The shell splits the file in the cat command whereas in the while loop the esearch command consumes the entire file, i.e. it doesn't care about the linebreaks. It's basically like this

[fhsantanna]

Great. So my fist code should be altered to the following one:

while read line <&3; do
esearch -db nucleotide -query "$line" | efetch -db nucleotide -format fasta
done 3<16S_list

It worked fine. Thank you very much.

[fhsantanna]

There is no empty line. I quoted, nothing happened. Neither do I...

[5heikki]

The second script works because the /dev/null redirect prevents esearch from consuming the entire file..

[fhsantanna]

Why does esearch consume whole file? Is it a bug? Because when I echo the lines of the list using while loop, everything works fine.

[5heikki]

It's not really a bug, but default behavior of the program..

Answer 1

[Michael]

A bit late to the party here but the esearch behavior is still the same. When using esearch in a loop with read, < /dev/null is required as the behaviour of esearch is to "slurp" all input from the input stream to the end. When reading a file with while read .... done < file.txt, the input stream is set to file.txt, therefore read gets to read only the first line in the stream, while esearch reads the rest without doing anything with it.