This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Manipulation Of Large Numbers Of Sequences Using Python Or Matlab

Hello,

I have millions of lines in the following format:

number sequence

The number refers to the frequency at which the sequence appears in my library and the sequence is as you would expect. A typical line might look like:

25565 AGTGCATTTTGGTTTAGGCATGA

Thus, this particular and fictitious sequence shows up 25565 times in my library.

I need to manipulate this data in the following way:

1) Confirm that the final 5 letters are correct (in this case, CATGA) and if not, remove the line.

and then

2) Remove the final 5 letters from all of the sequences on every line.

I have been trying to figure out how to load this information into either python as a dictionary or directly into matlab.

It would be very helpful to know whether this feat would be best approached with matlab, python, or something else. Also, how would it be best to load the data from the text file into a dictionary in python?

Thanks!

python matlab ngs
• 4,950 views • link
updated by Song Qiang • Song Qiang • written by zeropoint1 • zeropoint1

2 answers

If you have millions of lines, it is better to stream through the file one line at a time rather than to read the entire file into a data structure (python dictionary, array...). You can do this with something like:

inFile = open('inputFile.txt')
for line in inFile:
    data = line.strip().split()
    count = int(data[0])
    sequence = data[1]
    #do something with your count and sequence variables.

I don't understand what you mean by "Confirm the final 5 letters are correct"? What do you mean by correct?

• 235 views • link
written by Damian Kao Damian Kao

I guess he/she might be looking for a primer or TAG

• 0 views • link
written by Biojl Biojl

If the last 5 letters don't match an expected string, then the line must be discarded. This means that the sequences was misread and should not be considered. Thank you for your advice.

• 0 views • link
written by zeropoint1 • zeropoint1

If s is a string in python. Then the last five letters are just: s[-5:] So,

if (sequence[-5:] == 'CATGA'): #do something
• 360 views • link
written by KCC KCC

You may use a sed one-liner. Suppose the input file is in.txt and the output file is out.txt, run

sed -n '/CATGA$/ s/CATGA$//p' < in.txt > out.txt

• 153 views • link
written by Song Qiang • Song Qiang

Log in to answer this question.

More posts like this