As an example:
import sys
from Bio import SeqIO
# KMER size
SIZE = 11
recs = SeqIO.parse(sys.stdin, format="fasta")
for rec in recs:
for i in range(0, len(rec.seq)-SIZE):
seq = rec.seq[i:i+SIZE]
print (seq)
then you can:
# Get some protein data
wget -nc http://ftp.ensembl.org/pub/current_fasta/accipiter_nisus/pep/Accipiter_nisus.Accipiter_nisus_ver1.0.pep.all.fa.gz
# Chop it into pieces
gunzip -c Accipiter_nisus.Accipiter_nisus_ver1.0.pep.all.fa.gz | python chop.py > pieces.txt
# How many times did each piece occur
cat pieces.txt | sort | uniq -c | sort -rn > count.txt
looking at the end of the file:
tail count.txt
it prints:
1 AAAAAAAAAGH
1 AAAAAAAAAGD
1 AAAAAAAAAFH
1 AAAAAAAAAEK
1 AAAAAAAAADL
1 AAAAAAAAACR
1 AAAAAAAAAAX
1 AAAAAAAAAAN
1 AAAAAAAAAAM
1 AAAAAAAAAAD
the bottleneck is sorting, depending on the organism's size.
instead of sorting it as one file, split the counts.txt file into multiple files, sort each file in parallel, the merge sort the result