restructure rows and columns in perl or python (Interleaving columns by row pairs)

Question

[Ana]

Hi all, I have a SNPsfile (containing 11 millions SNPs) which I was using to create covariance matrix in Bayenv, so each column in this file corresponds populations and rows are SNPs, but for every SNP I have 2 rows (for two alleles), look like below (2 * nsnps "rows" and npops "columns"):

7        2       2       0        6        2       2
1        0       0       0        0        0       0
0        2       2       0        0        0       0
1        0       0       0        0        0       0

So in the example above I have 7 populations (columns) and 2 SNPs (rows). I need to modify the format of this file a bit. In the new file each row should correspond to one SNP and the number of columns should be twice the number of populations because each pair of numbers corresponds to each allele. So the new file should look like this ( nsnps "rows" and 2*npops "columns"):

7   1   2    0    2   0    0   0    6   0   2   0   2   0
0   1   2    0    2   0    0   0    0   0   0   0   0   0

I have Rcodes which do this manipulation job for me, but it seems that R is so slow, I just want to ask can anyone help me to figure out if there is anyway to do it in Perl or Python. I am new to both of them, I would appreciate any help to fix this issue. Thanks

[h.mon]

Can you show your R code and tell the size of your matrix and how much RAM you have available? Transposing a matrix should be quick in R, unless your matrix is too big and you are swapping to disk.

[WouterDeCoster]

It's not exactly transposing.

[h.mon]

You are right, it is not near transposing.

[WouterDeCoster]

Which is great because there is no need to load the entire matrix.

[ivivek_ngs]

is it me or someone who did not understand the output format. I feel so noob and still, cannot figure out what the OP wanted. Am glad Wouter figured it out but I would be glad if I can understand what the OP is trying to achieve. It will be nice to learn something new. :)

[ivivek_ngs]

Ah, I understood now the format and what the OP is trying to achieve. Actually, it is not replacing rows to the column to its entirety. If the moderator could help in changing the question else it will be misleading.

[WouterDeCoster]

I changed it to "restructure", can't think of anything more specific.

[ivivek_ngs]

yes, it is much better now and a reader will not be misled. Thanks, @Wouter. At least other readers will simply not copy the code rather read the query posted if they need any help with this post.

[h.mon]

Interleaving columns by row pairs? Interleaving columns by every two rows?

[WouterDeCoster]

That's a good one! ;)

Answer 1

[WouterDeCoster]

I think this should do the job:

Save as rearrangingalleles.py and execute as python rearrangingalleles.py myinput.txt > myoutput.txt

[Buffo]

More profesional use .write:

import sys
output = open('output.txt', 'w')
with open(sys.argv[1]) as input:
  while True:
    line1 = [item for item in input.readline().split()]
    if len(line1) == 0:
        break
    line2 = [item for item in input.readline().split()]
    output.write(' '.join([line1[i] + " " + line2[i] for i in range(len(line1))]) + '\n')
print '\n' + '\t' + 'Job completed!'

Save as rearrangingalleles.py and execute as python rearrangingalleles.py myinput.txt

[WouterDeCoster]

I firmly disagree.
It's a very convenient feature if scripts write to stdout, as such you can use them when piping. Also, it allows specifying both the output name and output directory.

While talking about more professional:

• didn't close the output file
• you should use the 'with' statement for opening files

e.g.:

with open(sys.argv[1]) as input, open('output.txt', 'w') as ouput:

[Buffo]

Also, it allows specifying both the output name and output directory.

prefix = sys.argv[1].split('.')[0]
output = open(prefix + '_output.txt', 'w')    #you will never have to specify output name or directory

you should use the 'with' statement for opening files

??? Why?

[WouterDeCoster]

It's nice that you defend your opinion, but I would suggest considering you might be wrong.

Your code will crash if the current directory is not writeable for the user, which is not an uncommon situation for directories with (shared) data files. Also, the user doesn't have the freedom to

• Stream the output to another command on stdin (fundamental concept in unix pipelines)
• Choose the output name and directory of their choice

With regard to "why using the with statement" see this page of Python for beginners.

[Buffo]

You are a very funny person WouterDeCoster :). I'm glad you're learning to program but, in addition to online courses I recommend using common sense, sometimes it is very useful!

Best.

[WouterDeCoster]

Due to the limited meta-communication online I'm not sure if you are just trolling me or are genuinely an arrogant fool. Anyway, thanks for the advice.

Have a nice day.

[Ana]

Thanks so much WouterDeCoster, it produced exactly the file I wished in less than 1 minute.

[WouterDeCoster]

If my answer resolved your question you can mark it as accepted.