This is a general question regarding the ShellCommandRequirement feature.
Would anyone happen to be able to explain why ShellCommandRequirement is required to use Pipes in the CWL arguments section?
The pipes will work if I use ShellCommandRequirement and set ShellQuote to false for the pipes.
However, the following CWL syntax looks like it should run fine based on the command line that is generated in the cwltool logging, but for some reason it seems the pipe is not working.
Arguments section:
arguments:
- /opt/common/CentOS_6-dev/vardict/v1.5.1/bin/VarDict
- -E
- $(inputs.column_for_region_end)
- -G
- $(inputs.reference_fasta)
- -N
- $(inputs.tumor_sample_name)
- -c
- $(inputs.column_for_chromosome)
- -b
- $(inputs.tumor_bam.path + '|' + inputs.normal_bam.path)
- -g
- $(inputs.column_for_gene_name)
- -f
- $(inputs.allele_freq_thres)
- -S
- $(inputs.column_for_region_start)
- -r
- $(inputs.min_num_variant_reads)
- $(inputs.bed_file)
- '|'
- /opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/testsomatic.R
- '|'
- /opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/var2vcf_paired.pl
- -N
- $(inputs.tumor_sample_name + '|' + inputs.normal_sample_name)
- -f
- $(inputs.allele_freq_thres)
Resulting command from cwltool logging:
[job vardict] /scratch/tmpbKz3yH$ /opt/common/CentOS_6-dev/vardict/v1.5.1/bin/VarDict \
-E \
3 \
-G \
/scratch/tmpcR4isu/stg9c4f932e-84fb-4108-99a2-627fd7aceec5/Homo_sapiens_assembly19.fasta \
-N \
C-4PU567-L002-d \
-c \
1 \
-b \
'/scratch/tmpcR4isu/stgeb5ea926-d425-468c-871a-1d593ae22d49/C-4PU567-L002-d_cl_aln_srt_MD_IR_FX_BR__aln_srt_IR_FX-duplex.bam|/scratch/tmpcR4isu/stgb8386cf2-387e-41b4-b999-3d9c9d60c9c1/C-4PU567-N002-d_cl_aln_srt_MD_IR_FX_BR__aln_srt_IR_FX-duplex.bam' \
-g \
4 \
-f \
0.0002 \
-S \
2 \
-r \
1 \
/scratch/tmpcR4isu/stg1e7697e8-7bd3-4e8e-a5fc-d5dea2b240f7/MSK-ACCESS-v1_0-probe-A.sorted.no_fp_snp.bed \
| \
/opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/testsomatic.R \
| \
/opt/common/CentOS_6-dev/vardict/v1.5.1/vardict_328e00a/var2vcf_paired.pl \
-N \
'C-4PU567-L002-d|C-4PU567-N002-d' \
-f \
0.0002 > /scratch/tmpbKz3yH/C-4PU567-L002-d.C-4PU567-N002-d.vardict.vcf
1 answer
Answered my own question after a bit of research:
ShellCommandRequirement in CWL allows us to use /bin/sh -c in order to pass our command as a string to a shell intrepreter. ShellQuote:false leaves shell metacharacters unquoted, and thus they can still be interpreted and used by a shell.
From CommandLineTool.job():
shellcmd, _ = self.get_requirement("ShellCommandRequirement")
if shellcmd is not None:
cmd = [] # type: List[Text]
for b in builder.bindings:
arg = builder.generate_arg(b)
if b.get("shellQuote", True):
arg = [shellescape.quote(a) for a in aslist(arg)]
cmd.extend(aslist(arg))
j.command_line = ["/bin/sh", "-c", " ".join(cmd)]
Without the ShellCommandRequirement the job will run with subprocess.Popen(shell=False) and thus no shell metacharacters can be used.
My misunderstanding was I thought that all CommandLineTools were run using a shell and could use metacharacters.
Glad it's helpful! It is also helpful for myself to be able to explain it. Another thing I realized was that even though the command is printed like that by cwltool when it runs the command, cwltool is a python process, and it's not the same as running the command from a shell / terminal outside CWL
Log in to answer this question.