This will make more sense with context so I'll explain.

I took about 8300 hits for endogenous retroviral LTRs from the human genome and retrieved 100bp flanks from either side of each hit for those that had any to put them together into 200bp preintegration sites. (I'm very sure the script for this works fine). I then took those 7326 flank/preintegration site sequences and carried out a local blast against the human genome (using UCSC hg38 reference genome) to identify any preintegration sites that may be in repetitive regions.

I don't know what kind of cutoff I should be looking for to identify "repetitive regions", but I started by filtering out queries that produced >1 hit as repetitive. This left me with only 19/7326 query (flank) sequences that were not considered repetitive. If I change the filter to remove any queries with >2 hits as repetive, I get 497/7326 non-repetitive flanks (still not a lot).

What's strange about all this is that out of all these sequences that are generated by cutting 100bp from either side of a hit in the human genome every single one generates at least one hit in the human genome.

Anyway here's the code I'm using to filter out flanks that produce either no hits (i.e. query id is not present in blast results) or fewer than 2 hits. The BLAST output format is outfmt=7 i.e. tabular with comments. There's a ton of useless script just to measure progress and a bunch of redundant measures of the same numbers just to make sure nothing was going wrong:

"""This script BLASTs every 200bp flank sequence against the human genome to check for matches. Any flanking sequences with fewer than 2 hits have their ids and hitcounts filtered into a new file."""

from Bio import SeqIO
import re
import time

print "Enter dataset"
dataset = raw_input()
startTime = time.time()

flanks = list(SeqIO.parse("flanks" + str(dataset) + ".fasta", "fasta"))
flanksnew = open("flanksnew" + str(dataset) + ".fasta", "w")
output = open("hitcount" + str(dataset) + ".txt", 'w')
outputall = open("hitcountall.txt", 'w')
blast = open("testing" + str(dataset) + ".txt")
lines = blast.readlines()
idlist = list("")
totalqueries_no = len(flanks)
queries_no = 0
nohitqueries_no = 0
lowhitqueries_no = 0
highhitqueries_no = 0
h=0
i=0
j=0
k=0

for index, line in enumerate(lines):
    if re.match("# Query:", line):
        hits = int(re.search(r'\d+', lines[index+3]).group()) 
        queries_no+=1
        qid = line[9:].rstrip()
        idlist.append(qid)
        print >> outputall, qid, hits
        if hits<3:
            lowhitqueries_no +=1
            print >> output, qid, hits
        else:
            highhitqueries_no +=1
    h+=1
    if len(lines)/h <= 10:
        i+=1
        print str(i*10) + "% finished (step 1)!"
        h=0     

print "Step 1 finished!"

for record in flanks:
    if record.id not in idlist:
        nohitqueries_no +=1
        print >> output, record.id, '0'
    j+=1
    if queries_no/j <= 10:
        k+=1
        print str(k*10) + "% finished (step 2)!"
        j=0

print "Done"

blast.close()
output.close()


print 'Total flanks:', queries_no, totalqueries_no, len(idlist)
#queries_no and totalqueries_no should not be identical if there are queries that did not produce hits, but they always are
print 'Total flanks with two or fewer:', lowhitqueries_no, nohitqueries_no
print 'Total flanks with more hits:', highhitqueries_no
print ('The script took {0} seconds!'.format(time.time() - startTime))

An example of final output (for a hit<3 filter)

Total flanks: 7326 7326 7326
Total flanks with two or fewer hits: 497 0
Total flanks with more hits: 6829

What am I doing wrong here? Either I'm doing something wrong while running the BLAST, the reference genome I'm using is wrong (I don't really know how much redundancy there is in different databases) or there's something wrong with this script. I think, at least.

However one problem I've already noticed with the results is that len(idlist) which consists of different query ids (qids) is 7326. That means each of these qids should have also been written into outputall (hitcountall.txt), but when I check it out there's only a very short list. So something is going wrong there.