I answered my own question using sort and python script (2.7.1). I will outline the basic logic I used.

Initially, sort the _export.txt files by cluster coordinate using the sort utility (GNU coreutils 8.4)

Then passed the sorted export files to a python script.

For each file:

 #Buffered file reader/line generator
    mygen = linesfromfile(infile)

    #Pre loop scan to first good (PF Y) and store as best line
    #This whole approach assumes a cluster coord sorted _export.txt file
    for ln in mygen:
        spl = ln.split('\t')
        rows = rows + 1
        if spl[21] =='Y\n':
            bst = spl
            break;

    #Loop to iterate through the lines
    for ln in mygen:
    #split line by tab delimiter
        spl = ln.split('\t')

        #Check if Pass Filter is Y
        if spl[21] == 'Y\n':

            #Compare cluster coords to previous coords.
            if (spl[4] == bst[4]) & (spl[5] == bst[5]):

                #same coords. Compare sum of base call scores
                if sumascii(spl[9]) > sumascii(bst[9]):

                    #replace best with current
                    bst = spl
            else:
                #Different coords. Indicates new set. 
                #Write input line to file, pass coordinate tuples to main thread.
                mainpipe.send( (bst[4],bst[5]) )
                outpf.write(ln)

                #Replace best line with current line.
                bst = spl
        rows = rows + 1

    #write final stored best line coord tuple
    mainpipe.send( (bst[4],bst[5]) )

The result of this is to get a set of unique coordinates from each file. Additionally, the logic removes optical duplicates by only writing the input lines that have the highest sum of base scores for each set of coordinates, and this is why the files have to be sorted by cluster coordinate first.

The next bit is to simply calculate the symmetric differences between the two sets of coordinate tuples.

#Get differences between coordinates of inputfile1 and inputfile2
symdiffs = tups1.symmetric_difference(tups2)

By getting the intersection of the coordinate tuple sets with the set of symmetric differences, we can find the unmated coordinate clusters in each file:

#Reduce tuple sets to those that are the difference entries
tups1.intersection_update(symdiffs)
tups2.intersection_update(symdiffs)

The length of the resulting sets will indicate the number of lines in the file that need to be removed into an unmatedpairsexport.txt file. This was accomplished with a second pass through the files.

The resulting files are then run though the maq script, fq_all2std.pl export2std, in order to get fastq files.

I don't have a file in this format so I can't recall the specifics.

One possible straightforward but memory hungry operation would be to load up both files into memory into two dictionaries that are keyed by the fields that you believe to be indicative of paired-ness. Then iterate over this dictionary keys and output the paired sequences ignoring the missing ones. I would guess that your memory requirements will be around 2-3 times the resulting fastq file size, but since you only have 67 million reads you might get by with say 4GB of ram.

If you don't know how to write such a script just edit your answer and paste in the first few lines of the export file and we'll give you a script that does the work. Please also comment on the edit if you do so, that way it shows up on my notifications.

Hello everyone, I need 2 or 3 files in Illumina Export Format to test a software function, (size does not matter). I've been looking through all over the web and have not been able to find them. (I understand SRA don't use it). I tried to find CASAVA old versions in order to build my own files, but the software is not available anymore. Could anyone provide a couple of files, a link or suggest a repository ? Thank you very much, ....

I am just trying to test an old software that worked very well over this format, I may try do something to make it work for current formats. Thanks