The last process of my Nextflow workflow failed with the following job.log. The workflow is run with a singularity container.

cp: error writing 'results/FILE_ST01_WGS_S1/S3TIMULATE_ST01_WGS_S1.sam': No space left on device
cp: cannot create regular file 'results/FILE_ST01_WGS_S1/FILE_ST01_WGS_S1.clean.bam': No space left on device
cp: cannot create regular file 'results/FILEST01_WGS_S1/FILE_ST01_WGS_S1.sorted.bam': No space left on device
cp: cannot create directory 'results/FILE_ST08_WGS_S8': No space left on device
...

It is a bit strange, since looking in the results directory, I could find all the directories and files which were created in previous processes. And all the previous Nextflow processes were completed without error (no errors in job.log(s)). The last process is also not very memory intensive. It is an rscript with takes some of the .txt and .tsv results from the outdir and makes a report.

There seems to be adequate space on the drive (it is a scratch drive).

The last process is to run a report which has little to do with the files and directories mentioned in the job.log errors.

The last process:

process run_report {

    tag "Run quality report"

    publishDir params.outdir, mode:'copy'

    input:
    tuple path(file_1), path(file_2)
    path outdir
    path rscript

    output:
    file('run_quality_report.html')

    script:
    """ 
    Rscript -e 'rmarkdown::render(input = "$rscript", output_dir = getwd(), params = list(directory = "$outdir"))'
    """
}

After inspecting the .command.run. It seems the process might be trying to copy ALL of the files in the outdir because outdir is an input in the 'nxf_stage()could this be the issue? And should I avoid using theoutdir` as an input?

nxf_stage() {
    true
    # stage input files
    rm -f Bam_stats_pf_Final.tsv
    rm -f Bam_stats_hs_Final.tsv
    rm -f results_victoria_all
    rm -f run_quality_report.Rmd
    cp -fRL /wynton/scratch/finterly/new_workflow/work/fe/ec06795d4f7811a7385efbc69cf4fa/Bam_stats_pf_Final.tsv Bam_stats_pf_Final.tsv
    cp -fRL /wynton/scratch/finterly/new_workflow/work/ce/9c20df52f840bb21f13c48dc4df75d/Bam_stats_hs_Final.tsv Bam_stats_hs_Final.tsv
    cp -fRL /wynton/scratch/finterly/results_victoria_all results_victoria_all
    cp -fRL /wynton/scratch/finterly/new_workflow/run_quality_report.Rmd run_quality_report.Rmd
}

nxf_unstage() {
    true
    cp .command.out /wynton/scratch/finterly/new_workflow/work/f5/85eaf31ed30707d936a7fe8a8c9841/.command.out || true
    cp .command.err /wynton/scratch/finterly/new_workflow/work/f5/85eaf31ed30707d936a7fe8a8c9841/.command.err || true
    cp .command.trace /wynton/scratch/finterly/new_workflow/work/f5/85eaf31ed30707d936a7fe8a8c9841/.command.trace || true
    [[ ${nxf_main_ret:=0} != 0 ]] && return
    IFS=$'\n'
    for name in $(eval "ls -1d run_quality_report.html" | sort | uniq); do
        nxf_fs_move "$name" /wynton/scratch/finterly/new_workflow/work/f5/85eaf31ed30707d936a7fe8a8c9841 || true
    done
    unset IFS
}