Biostar Beta. Not for public use.
Question: Calling Qiime scripts sequencially in Python 2.7 program
1
Entering edit mode

Hello all,

I'm beginning to learn Python 2.7 and need to develop some applications for a lab project.

I want to run an external program and then use its output as input for another external call in the same script. The issue is that the script tries to call the second program even before the first one has finished working, so the output is not yet available to be used as its input. How can I solve this problem?

Here is my code:

picker = Popen ([pick_open_reference_otus.py, -i, allseq_file, -r, refseq_file, -o, noName, -s, 0.1, -m, usearch61, -p, param_file ], shell = True)
picker_out = os.path.join(path, "otu_output/otu_table_mc2_w_tax.biom")

Thank you all!

ADD COMMENTlink 3.2 years ago uxue33 • 20 • updated 3.2 years ago WouterDeCoster 39k
Entering edit mode
1

I'm not 100% sure because I have not done any work with the subprocesses module, however I think you can just use the wait() module. something like picker.wait(picker_out) or something to that effect.

ADD REPLYlink 3.2 years ago
skbrimer
• 530
Entering edit mode
1

or picker_out = picker.wait(os.path.join(path, "otu_output/otu_table_mc2_w_tax.biom") sorry. I'm not sure about the syntax

ADD REPLYlink 3.2 years ago
skbrimer
• 530
Entering edit mode
1

last one :)

picker = Popen ([pick_open_reference_otus.py, -i, allseq_file, -r, refseq_file, -o, noName, -s, 0.1, -m, usearch61, -p, param_file ], shell = True)
picker.wait()
picker_out = os.path.join(path, "otu_output/otu_table_mc2_w_tax.biom")
ADD REPLYlink 3.2 years ago
skbrimer
• 530
Entering edit mode
0

Thank you a lot for your help!

ADD REPLYlink 3.2 years ago
uxue33
• 20
Entering edit mode
1

https://docs.python.org/2/library/subprocess.html

Subprocess has what you want.

Python can be rather clunky for this kind of thing. If you need data from some of these programs you're calling to end up being processed in python, it is better to pull the data into python from the "outside". You may want to explore other options for managing pipelines, there are plenty of posts and tutorials on here about it.

ADD REPLYlink 3.2 years ago
pld
4.8k
Entering edit mode
0

Thank you for the link!

ADD REPLYlink 3.2 years ago
uxue33
• 20
1
Entering edit mode

I used the following code for similar jobs:

retmode = subprocess.call(shlex.split('your shell command with arguments'))
if retmode > 0:
    sys.exit("Prinseq returned error code {} for sample {}".format(retmode, target))

I find the shlex module useful to structure command prior to using subprocess.call, it allows you to just write the string of the command and it will take care of the formatting in a list.

ADD COMMENTlink 3.2 years ago WouterDeCoster 39k
Entering edit mode
0

Than you a lot for your code!

ADD REPLYlink 3.2 years ago
uxue33
• 20
Entering edit mode
0

Dear all,

I found this post very useful and tried to use the solution myself. The thing is that I presume I get an error in the variable interpolation process that I can't figure out how to fix (I haven't used Python very often for this kind of task, because I usually use Perl for external program related pipelines). Here's my code and the error it produces.

import sys, os
import subprocess
import shlex

allseq_file = "/home/jllavin/Metagenomics/Test.fasta"
refseq_file = "/home/jllavin/Metagenomics/Dbs/99_otus.fasta"

picker = subprocess.call(shlex.split('pick_open_reference_otus.py -i allseq_file -r refseq_file -o output2 -s "0.1" -m "usearch61" -p parameters.txt -f'))
    if picker > 0:
    sys.exit[(picker)]

The error:

Traceback (most recent call last):
  File "/usr/lib64/python2.7/lib-tk/Tkinter.py", line 1470, in __call__
    return self.func(*args)
  File "QIIME_shlex.py", line 45, in button_ex_callback
    sys.exit[(picker)]
TypeError: 'builtin_function_or_method' object has no attribute '__getitem__'
ADD REPLYlink 3.2 years ago
JL
• 0
Entering edit mode
1

hi joluito,

for sure sys.exit[(picker)] is wrong. The square brackets don't serve a purpose here. Try sys.exit(picker). (This is just the return code of the script, in case something goes wrong)

for the rest, I don't think your -i allseq_file is going to work. Or is allseq_file an actual file in the current directory?

If it's a variable in your python code (what I suspect) you need to construct the string differently:

picker = subprocess.call(shlex.split('pick_open_reference_otus.py -i ' + allseq_file + ' -r ' + refseq_file + etc etc'))

As such the value of the variable is inserted in the string, fed to shlex.split() and used for the process.

Note the spaces!

It can be helpful to first construct your command:

cmd = 'pick_open_reference_otus.py -i ' + allseq_file + ' -r ' + refseq_file + etc etc'
print(cmd)
picker = subprocess.call(shlex.split(cmd))

and print that to see if everything is okay

ADD REPLYlink 3.2 years ago
WouterDeCoster
39k
Entering edit mode
0

Sorry, I was adding the rest of the code while you answered

;)

ADD REPLYlink 3.2 years ago
JL
• 0
Entering edit mode
1

So as I suspected, you are passing the variable's name to pick_open_reference_otus.py, rather than the variable's value. Is that clear?

ADD REPLYlink 3.2 years ago
WouterDeCoster
39k
Entering edit mode
0

Interesting point you raised here.

So, I'm not interpolating the variable's "string value" by passing its "name" to the command...

Therefore the way to proceed is to generate the command in a "print style" and then using it via subprocess.call(shlex.split)

Thank you very much for your patience, I will try it and come back with the outcome.

Best

ADD REPLYlink 3.2 years ago
JL
• 0
Entering edit mode
1

Yes, just concat the strings together with the variables and don't forget spaces.

ADD REPLYlink 3.2 years ago
WouterDeCoster
39k
Entering edit mode
0

It worked smoothly.

Thank you very much for your kind support.

Best

JL

ADD REPLYlink 3.2 years ago
JL
• 0

Login before adding your answer.

Similar Posts
Loading Similar Posts
Powered by the version 2.0