This solution also has the 64kb limitation, correct?
I am constructing a workflow that contains two steps:
- the output of the first step is stdout (compatible, I believe, with File type)
- the input of the second is a string
I am not able to modify these steps. How can I connect these two steps? Do I have to make an adaptor step?
3 answers
Here's the ExpressionTool version:
$ cat u.cwl
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: ExpressionTool
requirements: { InlineJavascriptRequirement: {} }
inputs:
a_File:
type: File
inputBinding:
loadContents: true
expression: |
${ return { "a_string": inputs.a_File.contents }; }
outputs:
a_string: string
And it's output
$ echo "Hello, World!" > message
$ cwltool u.cwl --a_File message
/home/michael/cwltool/env/bin/cwltool 1.0.20171107133715
Resolved 'u.cwl' to 'file:///home/michael/u.cwl'
{
"a_string": "Hello, World!\n"
}
Final process status is success
Yes, thank you for noticing that. The only way to get around the 64kb limit is with a dedicated CommandLineTool.
Which I've added an example at A: Converting stdout (File) to string
And a third way, that converts any size of File that will fit into system memory into a CWL string. This is achieved using the JSON feature from http://www.commonwl.org/v1.0/CommandLineTool.html#Output_binding
$ echo 'Hello,
World!' > message
Our input file
$ cat v.cwl
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
inputs:
a_File: File
baseCommand: python
arguments:
- prefix: -c
valueFrom: |
import json
a_string = ""
with open("$(inputs.a_File.path)", "r") as a_File:
a_string = a_File.read()
with open("cwl.output.json", "w") as output:
json.dump({"a_string": a_string}, output)
outputs:
a_string: string
The CWL CommandLineTool itself
$ cwltool ../v.cwl --a_File message
/home/michael/common-workflow-language/env/bin/cwltool 1.0.20171107133715
Resolved 'v.cwl' to 'file:///home/michael/v.cwl'
[job v.cwl] /tmp/tmp4YLbgd$ python \
-c \
'import json
import os
a_string = ""
with open("/tmp/tmp3xHM90/stgdc883f3a-b4e7-4eab-aa42-3cddb08a822b/message", "r") as a_File:
a_string = a_File.read()
with open("cwl.output.json", "w") as output:
json.dump({"a_string": a_string}, output)'
[job v.cwl] completed success
{
"a_string": "Hello,\nWorld!\n"
}
Final process status is success
And the CWL File to CWL string converter in action. As you can see, this reads the entire file into memory.
Hello eduardo,
If the stdout is less than 64kb then you can express it as a string either during the gathering of the outputs.
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool
inputs:
message: string
baseCommand: echo
arguments: [ $(inputs.message) ]
stdout: message
outputs:
message_File: # Just to illustrate that you can process a file multiple times, not necessary for message_string to work
type: File
outputBinding:
glob: message
message_string:
type: string
outputBinding:
glob: message
loadContents: true
outputEval: $(self[0].contents)
And the output:
$ cwltool ../t.cwl --message "Hello, world!"
/home/michael/cwltool/env/bin/cwltool 1.0.20171107133715
Resolved '../t.cwl' to 'file:///home/michael/t.cwl'
[job t.cwl] /tmp/tmpAljfTQ$ echo \
'Hello, world!' > /tmp/tmpAljfTQ/message
[job t.cwl] completed success
{
"message_string": "Hello, world!\n",
"message_File": {
"checksum": "sha1$09fac8dbfd27bd9b4d23a00eb648aa751789536d",
"basename": "message",
"location": "file:///home/michael/u/message",
"path": "/home/michael/u/message",
"class": "File",
"size": 14
}
}
Final process status is success
If you don't want to change your tool definition, then you can use an ExpressionTool to convert the File to a string.
Thanks! That's what I was looking for.
Log in to answer this question.
Please post your example code and expected output
Output from first step:
Input from second step:
Besides the tag it's unclear that this is about cwl - would be good to mention that. The question doesn't make sense if you don't read the tags.