Java memory issue in GATK

Question

[boymin2020]

I am using HaplotypeCaller of GATK to do variants calling for near 349 WGS human data, which means there are 349 jobs in the cluster. The following is my command.

java -Xmx4G \
-Djavaio.tmpdir=$tmp \
-jar $jar \
-T HaplotypeCaller \
-R $ref \
--genotyping_mode DISCOVERY \
-stand_call_conf 10 \
--emitRefConfidence GVCF \
--variant_index_type LINEAR \
--variant_index_parameter 128000 \
-D $DBSNP \
--comp:HapMapV3 $HpMpV3 \
-pairHMM VECTOR_LOGLESS_CACHING \
$infofields \
-log $log \
-I $bamFil \
-o $outVcf

An error came out for one job. It seems the cluster is short of memory. But I checked the memory usage, there was still many memory unused. Why it can not allocate memory to this job ?

Blockquote Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000007b5080000, 168820736, 0) failed; error='Cannot allocate memory' (errno=12)

[Asaf]

Did the specific node had more memory or the cluster in general? If the jobs are not distributed well enough, several jobs find themselves on the same node, exhausting its memory.

[boymin2020]

That is one kind of situation. But the job had run a long time and all of the nodes show very enough unused memory.