This is a test version of Biostars. For the public version, visit https://www.biostars.org.
best practice for renaming file in CWL workflow

I've been using this method to rename files at the end of a step:

  processed_fastq_1:
    type: File
    outputBinding:
      glob: ${ return '**/' + inputs.fastq1.basename }
      outputEval: |
        ${
          self[0].basename = inputs.add_rg_SM + '_R1.fastq.gz';
          return self[0]
        }

But I'm not sure if this is an acceptable method. Should there be an intermediate step using python or calling to a script to rename files in the middle of a CWL workflow?

cwl
• 5,068 views • link
updated by Michael R. Crusoe Michael R. Crusoe • written by ionox0 ionox0

I think either method would be acceptable and could come with their own benefits/pitfalls. For example, one benefit of using an intermediate renaming step is that it would be compatible with a greater number of execution engines such as Cromwell, cwlexec, snakemake etc.. By creating an intermediate step you might eliminate some of the issues that the engine would impose. I am speaking only from experience with CWLExec, but I know that command would not be interpreted by cwlexec for Job Submissions to a cluster.

The obvious benefit of this is if you have an engine from which you know it works, you save an unnecessary intermediate step.

• 1 views • link
written by drkennetz drkennetz

I have been working with IBM quite a bit on bug fixes in regards to what they support and don't support. I work with CWLexec every day in our lab to develop pipeline automation. In theory, they support InlineJavascriptRequirement and ShellCommandRequirement but in practice CWLexec is quite picky. If you do not use cwlexec this convo doesn't really matter :).

• 1 views • link
written by drkennetz drkennetz

Is there any way to do this after the fact in an ExpressionTool?

• 0 views • link
written by alanh alanh

2 answers

Hello ionox0,

That is acceptable way to rename, yes. However, ** is not part of the POSIX glob definition, so you'll need to amend that. Perhaps the following simplification works with your tool?

  processed_fastq_1:
    type: File
    outputBinding:
      glob: */$(inputs.fastq1.basename)
      outputEval: |
        ${
          self[0].basename = inputs.add_rg_SM + '_R1.fastq.gz';
          return self[0]
        }

It has been requested that there be an easier way to rename output Files but there is no proposed new syntax yet. Perhaps you have a suggestion?

https://github.com/common-workflow-language/common-workflow-language/issues/668

The same link shows how to rename a File between steps.

• 1 views • link
written by Michael R. Crusoe Michael R. Crusoe

Thanks very much @mr-c!

My suggestion for changing the output file name would be something like:

outputs:
  out_thing:
    glob: *.fastq
    newName: my_new_name.fastq

If I have time I could put more research into how to implement this feature.

• 1 views • link
written by ionox0 ionox0

Thank you! The following syntax works:

  processed_fastq_1:
    type: File
    outputBinding:
      glob: $('*/' + inputs.fastq1.basename)
      outputEval: |
        ${
          self[0].basename = inputs.add_rg_SM + '_R1.fastq.gz';
          return self[0]
        }
• 1 views • link
written by ionox0 ionox0

I would suggest adding a simple mv command to the end of the command line, via arguments, e.g. 

arguments:  
  - position: 1000  
    shellQuote: false  
    valueFrom: ' && mv $(inputs.fastq1.basename) $(inputs.add_rg_SM)_R1.fastq.gz'

then you could just glob $(inputs.add_rg_SM)_R1.fastq.gz

This way you don't create duplicate files, which would be the case when using a separate step. Also, other users using the tool can see that renaming is happening just by looking at the command-line.

• 1 views • link
written by bogdan.gavrilovic bogdan.gavrilovic

Yes, this looks like a good solution, and is perhaps better practice than modifying the basename of the output itself

I also wonder if a new cwl step entirely would be an even more robust / cleaner solution

• 0 views • link
written by ionox0 ionox0

Log in to answer this question.

More posts like this