This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Issue assigning unique ID's to a CWL pipeline

Hi everyone,

I have been playing around with CWL and am currently stumped with the unique identifier option. I have assigned an unique identifier to one of my input files in my pipeline, which runs successfully. However, how can I use this ID to track provenance? Is there some way that CWL permanently associates a file with an ID, so that at the end of a workflow, the user can identify which ID's were associated with a given file(s)? Or CWL only accepts a file as input with a specific ID?

I have confused with the use of unique ID's in CWL and would appreciate if someone could clarify this for me.

Thank you for the help.

cwl

An Identifier in a CWL document is only unique in the context of the document you define it in. Only (links to) the files themselves are passed around between workflow steps. The identifier used during execution of a step is void after the step has been completed.

Can you give an example of what you have done? You are speaking about a unique identifier "option". My understanding is that IDs for files (or other input or output objects) are not optional in CWL.

1 answer

The location of a File object (e.g. in an input JSON/YAML document) will be the identifier URI for that file, and you can use path to give it a different file system location, or the contents key can be used to provide the file content as in-line string (for small files).

So the simplest is if location is a downloadable URI for the file.

These File properties will be propagated from a step to the workflow output JSON document, but may be lost if going to an intermediate step.

We faced this challenge in CWLProv, as typical use of CWL will mean the file location is just derived from the path, e.g. a URI like file:///home/stain/data1.csv which is obviously not globally unique or particularly permanent.

If you use --provenance option with cwltool you can generate the CWLProv research object that will include the trace of intermediate values and snapshot copies of the data files.

You can see that we generate a new @id property with a UUID like urn:uuid:071d1b5c-1b2b-4995-a6e0-80821af85abd identifying the file as it is at time of running the workflow, thus having a fixed checksum and file content.

In the PROV trace we link this identifier with the byte-snapshot (here checksum b9214658cc453331b62c2282b772a5c063dbd284) and it's appearance as being generated as an output or used as an input of different steps, according to the W3C PROV model

For cwltool, as a reference implementation that can be run anywhere, we have no idea where the data live or will live in the future, so UUID is the safest way to make a unique identifier - but In this code we will respect any existing @id, so you could use that to provide a better identifier for the --provenance subsystem, and I believe that unlike location it will be ignored/passed along by the rest of the CWL engine.

Log in to answer this question.