This is great! I didn't know about linkMerge, wish i would have learned this sooner.
Hi all, I have a cwl workflow where step1 outputs an array of items file. In step 2 I would like to add a file or an array of files to the array that is output from step1. Here is my example workflow:
cwlVersion: v1.0
class: Workflow
requirements:
- class: ScatterFeatureRequirement
- class: StepInputExpressionRequirement
- class: InlineJavascriptRequirement
inputs:
file1: File[]
file2: File[]
input_added_in_step2: File
string_input: string_input
steps:
step1:
run: step1.cwl
scatter: [file1, file2]
scatterMethod: dotproduct
in:
file1: file1
file2: file2
out:
[output_array]
step2:
run: step2.cwl
in:
input_array:
source: step1/output_array
valueFrom: $(self.concat(inputs.input_added_in_step2))
string_input: string_input
out: [output_array_and_file_added_and_rename]
outputs:
output_array_plus_file:
type: File[]
outputSource: step2/output_array_and_file_added_and_rename
The biggest issue here is that the workflow actually runs to completion, but it doesn't ever add input_added_in_step2 to the array created by step1. It just returns the array created by step1 with the string addition.
Thanks
2 answers
In the valueFrom: $(self.concat(inputs.input_added_in_step2)) expression, inputs will reference the inputs of the workflow step, not the input section at the beginning of the workflow document. Looking at the valueFrom section in this segment of the specifications leads me to believe there is no way to reference input_added_in_step2 in the StepInputExpression. You will need to define the file as input for step2 or create a separate step (presumably an expressionTool) with the necessary inputs to prepare the array for step2.
A similar problem was discussed here, maybe the examples can help: Combine subject with sample type in a scatter?
You can combine multiple inbound inputs using https://www.commonwl.org/v1.0/Workflow.html#MultipleInputFeatureRequirement
Try this:
cwlVersion: v1.0
class: Workflow
requirements:
- class: ScatterFeatureRequirement
- class: StepInputExpressionRequirement
- class: MultipleInputFeatureRequirement
inputs:
file1: File[]
file2: File[]
input_added_in_step2: File
string_input: string_input
steps:
step1:
run: step1.cwl
scatter: [file1, file2]
scatterMethod: dotproduct
in:
file1: file1
file2: file2
out:
[output_array]
step2:
run: step2.cwl
in:
input_array:
source: [ step1/output_array, input_added_in_step2 ]
linkMerge: merge_flattened
string_input: string_input
out: [output_array_and_file_added_and_rename]
outputs:
output_array_plus_file:
type: File[]
outputSource: step2/output_array_and_file_added_and_rename
Thanks Tom, maybe you could help us write a section in the CWL user guide about this?
https://github.com/common-workflow-language/user_guide/issues/103
You don't have to make a perfect lesson, even just writing what you can is helpful; we'll clean up and format as necessary :-)
The merge_flattened is actually something I found on another one of your posts that ended up solving my problems:
CWL: combine two arrays as workflow step input
Super helpful for me!
I would be very happy to do so. Since i have to submit my master's thesis this month i'm a little stressed out and won't be able to take care of it right now. I have made a calendar entry to make sure i won't forget.
I spent a lot of time reading and writing CWL documents in the past months. A more in-depth user guide would have saved me some some long hours of being confused. It's great that you are expanding it!
*edit: Learning about linkMerge has just allowed me to purge an abhorrent ExpressionTool from one of my workflows, which makes me very happy :)
Congratulations on your pending thesis! Yes, no rush. Any contribution is appreciated!
Log in to answer this question.