CWL in toil

Question

[a.james]

Dear All,

I am using CWL workflow in toil. And it is not recognising the path environment, which otherwise is fine. I mean, if I run the workflow as cwltool it is finishing successfully. When I try it in Toil it is not recognising the path env. The command line tool: sp.cwl

    cwlVersion: v1.0
    class: CommandLineTool
    doc: Spladder

    baseCommand: [python, -m, spla.spla]

    requirements:
     - class: InlineJavascriptRequirement
     - class: InitialWorkDirRequirement
       listing:
        - entry: "$({class: 'Directory', listing: []})"
          entryname: $(inputs.split_outDir)
          writable: true
     - class: EnvVarRequirement
       envDef:
        - envName: PYTHONPATH
          envValue: $(inputs.spl_pythonpath)

 inputs:
    ......
     split_outDir:
      type: string
      spl_pythonpath:
       type: string

output:
spla_out:
  type: File
  outputBinding:
   glob: $(inputs.split_outDir)/splr/*.pi

The YML file:

    spl_pythonpath: /cluster/home/user/spl/python 
    split_outDir: splice

The workflow which I used to call the above commandline tool,

    #!/usr/bin/env cwl-runner
    cwlVersion: v1.0
    class: Workflow

    requirements:
     - class: ScatterFeatureRequirement
     - class: SubworkflowFeatureRequirement
     - class: InlineJavascriptRequirement
     - class: MultipleInputFeatureRequirement

    inputs:
     .......
..........

    outputs:
     spla_dir:
       type: Directory
       outputSource: collect/spla_dir

    steps:
      spl:
        run: sp.cwl
        scatter: [spla_bam]
        scatterMethod: dotproduct
        in:
       ........
..................
         split_outDir: spladder_outDir
         spl_pythonpath: spl_pythonpath
        out: [spla_out]
      collect:
        in:
          spla_bam_files:
            source: [spl/spla_out]
            linkMerge: merge_flattened
        out: [spla_out]
        run:
          class: ExpressionTool
          id: "collect_step"
          inputs:
            spladder_bam_files: File[]
          outputs:
            spla_dir: Directory
          expression: |
           ${
            return {
              "spla_dir": {
                 "class": "Directory",
                 "basename": "Splicing/spladder",
                 "listing": [].concat.apply([], inputs.spladder_bam_files)
              }
            };
            }

Then I used Toil to run the above workflow as,

(toil-cwl-runner --stats --clusterStats --retryCount=0 --batchSystem=lsf --disableCaching --tmpdir-prefix ${TMP_DIR} --tmp-outdir-prefix ${TMP_OUT_DIR} --workDir ${WORK_DIR} --realTimeLogging --cleanWorkDir=never --clean=never --outdir ${OUT_DIR} --logDebug  --logFile ${OUT_DIR}/CWL.log --jobStore ${JOB_STORE} pipeline_main_parallel.cwl pipeline_main_example.yml)>& CWL.${DATE}.log &

Which is throwing error at,

330756-WARNING:toil.leader:E/b/jobUIrJ3T        y
330799-WARNING:toil.leader:E/b/jobUIrJ3T    /cluster/home/user/software/anaconda/envs/py2/bin/python2.7: can't open file '/cluster/home/user/spl/python/spla.py': [Errno 2] No such file or directory

The tool is using 10 parameter and the after the 10th parameter the python path is defined in inputs. Toil is not running it as it is defined in base command toil should run it as python -m spla.spla within the defined path.

[Michael R. Crusoe]

Hello a.james.

Does it work when you use the CWL reference runner, cwltool?

Also, neither cwltool nor toil support conda for installation. We both recommend the use of a fresh Python virtualenv. That may fix your issue.