Hi,I run the cwl workflow in the local machine. I write the cwl and run in [/Bigdata], but the running work seems to fill the /dev and reports that No space left on device. I want to konw why？ my fastq is about 70G
Filesystem Size Used Avail Use% Mounted on
udev 32G 4.0K 32G 1% /dev
tmpfs 6.3G 1.1M 6.3G 1% /run
/dev/vda1 99G 94G 0 100% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 32G 0 32G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/vdc1 1.4T 1007G 302G 77% /var/lib/docker
/dev/vdb1 296G 49G 232G 18% /home
overlay 1.4T 1007G 302G 77%
Filesystem Size Used Avail Use% Mounted on
/dev/vdd1 2.0T 220G 1.7T 12% /Bigdata
2 answers
Your cwl-runner is writing everything to a temporary directory before collecting the final output and placing it in /Bigdata.
The solution depends on which cwl-runner you are using. For cwltool, you could use the command line parameters --tmpdir-prefix and --tmp-outdir-prefix and specify a device with lots of room.
Depending on your os you could probably also bind the tmp directory to another drive with a command like mount -B /tmp /mnt/tmp.
Probably because of a bunch of temporary files going to /tmp/. Samtools has a -T argument to specify where temporary files should be made.
Wouter is absolutely right. Just for you know what to look for next time, this most likely where your /temp is:
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 99G 94G 0 100% /
You're analysis shows you it's completely filled up. Something like this is always a warning sign and is very likely to cause system instability.
Thank you. my error is :
samtools sort: failed to create temporary file "/BOredq/NA12878.sorted.bam.tmp.0000.bam": No space left on device
And samtools sort module's "-T" option just specify the prefix instead of output dir. $samtools sort
Usage: samtools sort [options...] [in.bam]
Options:
-l INT Set compression level, from 0 (uncompressed) to 9 (best)
-m INT Set maximum memory per thread; suffix K/M/G recognized [768M]
-n Sort by read name
-t TAG Sort by value of TAG. Uses position as secondary index (or read name if -n is set)
-o FILE Write final output to FILE rather than standard output
-T PREFIX Write temporary files to PREFIX.nnnn.bam
--input-fmt-option OPT[=VAL]
Specify a single input file format option in the form
of OPTION or OPTION=VALUE
-O, --output-fmt FORMAT[,OPT[=VAL]]...
Specify output format (SAM, BAM, CRAM)
--output-fmt-option OPT[=VAL]
Specify a single output file format option in the form
of OPTION or OPTION=VALUE
--reference FILE
Reference sequence FASTA FILE [null]
-@, --threads INT
Number of additional threads to use [0]
can we specify all of cwl output dir including tmp files to running dir such as [ /Bigdata] ?
And samtools sort module's "-T" option just specify the prefix instead of output dir.
And what if you would use, as a "prefix" /Bigdata/tmp/mytemp?
Log in to answer this question.
