This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Converting stdout (File) to string

I am constructing a workflow that contains two steps:

  1. the output of the first step is stdout (compatible, I believe, with File type)
  2. the input of the second is a string

I am not able to modify these steps. How can I connect these two steps? Do I have to make an adaptor step?

cwl

Please post your example code and expected output

Output from first step:

outputs:
    accessions:
        type: stdout

Input from second step:

inputs:
    sequences:
        type: string

Besides the tag it's unclear that this is about cwl - would be good to mention that. The question doesn't make sense if you don't read the tags.

3 answers

Here's the ExpressionTool version:

$ cat u.cwl 

#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: ExpressionTool

requirements: { InlineJavascriptRequirement: {} }

inputs:
  a_File:
    type: File
    inputBinding:
      loadContents: true

expression: |
  ${ return { "a_string": inputs.a_File.contents }; }

outputs:
  a_string: string

And it's output

$ echo "Hello, World!" > message
$ cwltool u.cwl --a_File message 
/home/michael/cwltool/env/bin/cwltool 1.0.20171107133715
Resolved 'u.cwl' to 'file:///home/michael/u.cwl'
{
    "a_string": "Hello, World!\n"
}
Final process status is success

This solution also has the 64kb limitation, correct?

Yes, thank you for noticing that. The only way to get around the 64kb limit is with a dedicated CommandLineTool.

And a third way, that converts any size of File that will fit into system memory into a CWL string. This is achieved using the JSON feature from http://www.commonwl.org/v1.0/CommandLineTool.html#Output_binding

$ echo 'Hello,
World!' > message

Our input file

$ cat v.cwl 
#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool

inputs:
  a_File: File

baseCommand: python
arguments:
 - prefix: -c
   valueFrom: |
     import json
     a_string = ""
     with open("$(inputs.a_File.path)", "r") as a_File:
         a_string = a_File.read()
     with open("cwl.output.json", "w") as output:
         json.dump({"a_string": a_string}, output)

outputs:
  a_string: string

The CWL CommandLineTool itself

$ cwltool ../v.cwl --a_File message
/home/michael/common-workflow-language/env/bin/cwltool 1.0.20171107133715
Resolved 'v.cwl' to 'file:///home/michael/v.cwl'
[job v.cwl] /tmp/tmp4YLbgd$ python \
    -c \
    'import json
import os
a_string = ""
with open("/tmp/tmp3xHM90/stgdc883f3a-b4e7-4eab-aa42-3cddb08a822b/message", "r") as a_File:
    a_string = a_File.read()
with open("cwl.output.json", "w") as output:
    json.dump({"a_string": a_string}, output)'
[job v.cwl] completed success
{
    "a_string": "Hello,\nWorld!\n"
}
Final process status is success

And the CWL File to CWL string converter in action. As you can see, this reads the entire file into memory.

Hello eduardo,

If the stdout is less than 64kb then you can express it as a string either during the gathering of the outputs.

#!/usr/bin/env cwl-runner
cwlVersion: v1.0
class: CommandLineTool

inputs:
  message: string

baseCommand: echo

arguments: [ $(inputs.message) ]

stdout: message

outputs:
  message_File:  # Just to illustrate that you can process a file multiple times, not necessary for message_string to work
    type: File
    outputBinding:
      glob: message

  message_string:
    type: string
    outputBinding:
      glob: message
      loadContents: true
      outputEval: $(self[0].contents)

And the output:

$ cwltool ../t.cwl --message "Hello, world!"
/home/michael/cwltool/env/bin/cwltool 1.0.20171107133715
Resolved '../t.cwl' to 'file:///home/michael/t.cwl'
[job t.cwl] /tmp/tmpAljfTQ$ echo \
    'Hello, world!' > /tmp/tmpAljfTQ/message
[job t.cwl] completed success
{
    "message_string": "Hello, world!\n", 
    "message_File": {
        "checksum": "sha1$09fac8dbfd27bd9b4d23a00eb648aa751789536d", 
        "basename": "message", 
        "location": "file:///home/michael/u/message", 
        "path": "/home/michael/u/message", 
        "class": "File", 
        "size": 14
    }
}
Final process status is success

If you don't want to change your tool definition, then you can use an ExpressionTool to convert the File to a string.

Thanks! That's what I was looking for.

Log in to answer this question.