This is a test version of Biostars. For the public version, visit https://www.biostars.org.
Singularity image is not working in nextflow

Hello everyone,

I'm having trouble getting Singularity to work with my custom bcftools pipeline in Nextflow.

I've tested my pipeline with Conda and it works great. However, when I install software locally (instead of using Conda), it runs approximately 10 times faster due to the large number of times the Conda environment needs to be invoked.

To test Singularity, I followed these steps:

  1. downloaded bcftools singularity image from: https://cloud.sylabs.io/library/weizhu365/mocca-sv/bcftools_1-9
  2. tested it with singularity run bcftools_1-9_1.0.0.sif
  3. added necessary lines to nextflow.config: 
    process.container = '/path/to/bcftools_1-9_1.0.0.sif'
singularity.enabled = true

However, it doesn't seem to recognize the image. I've successfully run other Nextflow pipelines with Singularity, such as nf-core's nextflow run nf-core/sarek -profile test,singularity -r 2.7.1.

Do I need to take any additional steps? Any help would be greatly appreciated.

nextflow singularity
• 4,509 views • link
updated by ATpoint ATpoint • written by optimistsso4co3 • optimistsso4co3

It should work. At least this works:

$ cat main.nf 
nextflow.enable.dsl = 2

process Test {

  output:
  stdout

  script:
  """
  bcftools --version
  """

}

workflow { Test() }  

$ cat nextflow.config 
process.container = "/home/a/a_toen03/bcftools_1.17--haef29d1_0.sif"
singularity.enabled = true

...correctly running the bcftools command. Please elaborate on what exactly goes wrong. What is the error message (if any)? Can you try with the container I linked? Get via singularity pull docker://quay.io/biocontainers/bcftools:1.17--haef29d1_0 from quay.

• 0 views • link
written by ATpoint ATpoint

The command bcftools --version is functioning properly. However, when attempting to use the command bcftools view -h file.vcf.gz to view a specific file in the current directory, an error message is generated indicating that the file does not exist, even though it's in the working directory. In fact, no file can be detected when singularity image is specified.

Can you please test if this issue is present in your system?

My singularity version: 3.4.1-4.2.ohpc.1.3.9

• 0 views • link
written by optimistsso4co3 • optimistsso4co3

So the issue is that the container somehow does not have an access to file.vcf.gz? If you specified file.vcf.gz as nextflow's input you should have a softlink to that file in the nextflow work subdirectory. You may just get into the container singularity shell your_container.sif and check what is visible from there (ls, df)

• 0 views • link
written by Darked89 Darked89

I tested singularity shell your_container.sif inside the Work subdirectory and, even though file.vcf.gz is visible (ls -l), it can not be red when your_container.sif is activated

less: can't open, no such file or dir

On the other hand, process files (e.g. .command.sh) can be red

• 0 views • link
written by optimistsso4co3 • optimistsso4co3

That's not a helpful diagnostic as you need to mount folders if running singularity manually -- something that nextflow does for you. Again, please provide a reproducible example that includes some minimal testing data and the Nextflow code. Singularity is used with Nextflow by thousands of users, so it must be something with how you read/stage/provide the input data.

• 0 views • link
written by ATpoint ATpoint

Please provide a reproducible example for debugging.

• 0 views • link
written by ATpoint ATpoint

0 answers

No answers yet.

Log in to answer this question.

More posts like this