In this forum, it is a frequent pitfall to use grep only with -f. I used to give my answer Retrieve Single Vcf From List Of Snp Rs#, but I feel it would be better to expand that answer a little bit.

grep has an option -f. According to its manpage: -f FILE: Obtain patterns from FILE, one per line. Each line is interpreted as a regex pattern. I speculate in that case, grep -f will load all the patterns in RAM and for each input line it attempts each pattern in turn. Thus if there are 1000 lines in the pattern file, grep -f will be 1000 times as slow as grepping one pattern only (i.e. linear in the number of patterns).

grep also has an option -F, which is equivalent to fgrep. fgrep uses the Aho–Corasick algorithm, an algorithm that constructs one DFA for all the pattern strings. When grep push an input line through the DFA, all patterns are attempted at the same time. Thus the speed of grep -F is largely independent of the number of patterns (i.e. constant to the number of patterns). For fixed string matching, we should use grep -Ff; otherwise the speed will be extremely slow (see the benchmark in Retrieve Single Vcf From List Of Snp Rs#). In addition, it is recommended to add option -w to reduce false matching.

I have fallen to this pitfall for many years myself. I used to ask my colleagues not to use grep -f because it is slow. But one day, I suddenly felt something wrong: Aho–Corasick algorithm should be as fast as grepping one pattern; why grep -f is so slow? When I read the manpage, it became obvious that I was misusing grep for a long time. On the other hand, I have to say the design of both -f and -F is really confusing.

EDIT: the approximate answer is:

grep -wFf 2.txt 1.txt > 3.txt

which works most of time, but may have false positives in rare cases where the patterns appear in other columns. The accurate answer is:

awk 'BEGIN{while((getline<"2.txt")>0)l[$1]=1}l[$1]' 1.txt > 3.txt

which works all the time.

Just to add another potential solution [making the assumption that we are continuing to live in a UNIX world]:

Command:

sort -k 1 1.txt | join 2.txt -

Result:

gi|93045967|gb|CJ563048.1|CJ563048 gi|225427470|ref|XP_002270678.1| 63.27 98 34 1 617 330 124 221 3e-48 141

HTH

If 2.txt has more than one entry it is necessary to also sort it by its key (e.g. first column) by executing

sort -k 1 2.txt > 2.sorted.txt

and modifying above command to:

sort -k 1 1.txt | join 2.sorted.txt -

Here is one solution (edit: see also Retrieve Single Vcf From List Of Snp Rs# for an explanation of -F [fgrep]):

fgrep -f 2.txt 1.txt > 3.txt

Thank you .I have get the answer fgrep -w -Ff 2.txt 1.txt > 3.txt