More concise way to pass many parameters through to subworkflows

I have a lot of parameters in a pipeline, and would like to pass them down through several layers of subworkflows so the system is modular and reusable for different analyses, but it is cumbersome to have to list all of the parameters for each workflow, and also to change all of occurrences of it when one changes.

Is there a way to pass a group of parameters through to a subworkflow, but without explicitly listing all of them?

For example:

(inputs.yaml)

a_bunch_of_params_for_step_1:
  param_1: ...
  param_2: ...
  param_3: ...
  param_4: ...
  param_5: ...
  ...

a_bunch_of_params_for_step_2:
  param_1: ...
  param_2: ...
  param_3: ...
  param_4: ...
  param_5: ...
  ...

(workflow.yaml)

inputs: 
  a_bunch_of_params_for_step_1: (record type?)
  a_bunch_of_params_for_step_2: (record type?)

steps:
  step_1:
    inputs: 
      inputs_1:  a_bunch_of_params_for_step_1

  step_2:
    inputs: 
      inputs_1:  a_bunch_of_params_for_step_2

It seems like this might be possible with record types, but I haven't seen any examples where the fields in the record aren't all explicitly specified as a part of the record.

cwl

• 1,713 views • link

updated 8.3 years ago by kaushik.ghose

• written 8.4 years ago by ionox0

1 answer

One solution, not a very elegant one, would perhaps be to supply a text file (say a YAML file) with all the parameter (name, value) pairs, as a shared input to each tool. Each tool then has javascript in the CWL that grabs the parameters it requires (use load_contents to read the file contents into an accessible variable) and passes them appropriately to the command line.

But this is an interesting use case - a common group of parameters consumed by a workflow of which subsets are consumed by parts of the workflow. We should keep this in mind.

Thanks -Kaushik

• 0 views • link

written 8.3 years ago by kaushik.ghose

It's just easier to group the parameters into one input so that you don't have to list them all in each inputs section. It's particularly cumbersome when there are several layers of subworkflows, and all you really want to do is just get the inputs to the underlying tool.

I found that it is indeed possible to pass record types through from workflow to subworkflow, which solves most of the issue. The other part of the issue is getting only the fields from the record that you want for a particular workflow step, and that can be solved with valueFrom:

in:
  run_tools: run_tools
  java:
    valueFrom: $(inputs.run_tools.java_7)
  arrg:
    valueFrom: $(inputs.run_tools.arrg_path)

Which may not be the best solution. I'm curious what is the load_contents feature you're describing?

• 1 views • link

written 8.3 years ago by ionox0