This would work:

git clone https://github.com/BioInf-Wuerzburg/SeqFilter.git
cd SeqFilter
make  # just fetches some libraries, no root or anything required

bin/SeqFilter big.fasta --ids idx.txt --ids-exclude --out big-filtered.fasta

Simple way is to get a list of IDs that you would like to fetch from fasta. This could be done with 'grep' .

grep "^>" input.fasta | sed 's/>//' | grep -v - -f Ids.txt > retreive_IDs.txt

Then you could use something like pyfaidx or samtools

samtools faidx input.fasta `cat retreive_IDs.txt` 

Boy, this really comes up a lot. Using the BBMap package:

filterbyname.sh in=file.fasta out=filtered.fasta names=names.txt include=f

You could execute an awk statement that excludes a set of sequence IDs:

$ awk '!/\<idA\>/ && !/\<idB\>/ && ... && !/\<idN\>/' RS='>' ORS='>' seqs.fa > filtered_seqs.fa

Each exclusion pattern looks like:

!/\<idA\>/

The exclamation point excludes whatever matches the inside of the pattern. Inside the pattern is the sample ID value idA, which is wrapped in word boundary markers \< and \> to enforce an exact match on the string idA. This avoids excluding records with partial ID matches.

The RS value separates input by record via the > character. The ORS value prints output with the > as a separator.

While it might look a bit convoluted, it's a pretty straightforward awk call. It would be easy to write a bash, Perl or other shell script to construct this statement, given a text file of IDs you need to exclude.

Since you're streaming through the file one record at a time (or one small buffer containing a few records), and only reading the 20 GB file once, awk should use very little memory and run very quickly.

Other approaches may require preprocessing a very large file into something else, indexing it, or reading large chunks of the file into memory for parsing, which could take a while with a 20 GB input.