I have solved my problem just forcing the use of conda with Snakemake. But out of curiosity I have made some test, and found slightly different version of conda in different nodes.
Test of the version of the software:
Single node:
conda -V
conda 23.3.1
mamba -V
conda 23.3.1
slurm script:
conda -V
conda 23.9.0
mamba -V
conda 23.3.1
Try to make a new environment:
mamba env create --quiet --name testenv
EnvironmentFileNotFound: '/mnt/shared/home/usr/environment.yml' file not found
In the .bashrc file:
# >>> conda initialize >>>
# !! Contents within this block are managed by 'conda init' !!
__conda_setup="$('/home/usr/miniforge3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
if [ $? -eq 0 ]; then
eval "$__conda_setup"
else
if [ -f "/home/usr/miniforge3/etc/profile.d/conda.sh" ]; then
. "/home/usr/miniforge3/etc/profile.d/conda.sh"
else
export PATH="/home/usr/miniforge3/bin:$PATH"
fi
fi
unset __conda_setup
if [ -f "/home/usr/miniforge3/etc/profile.d/mamba.sh" ]; then
. "/home/usr/miniforge3/etc/profile.d/mamba.sh"
fi
# <<< conda initialize <<<
Can you check if the conda/mamba executables you use are the same on both the node you ran things successfully on and on the compute node?
Also, your sysadmin is in a much better position to help you than we are.
i bet there is some script in your .bashrc (or the one you inherited from root) that is putting you in a hostile conda environment when snakemake dispatches to that worker node.
i would try ssh'ing directly to any node and try to run some commands:
mamba env create --quiet --name testenvI'm adding 'hostile conda environment' to my normal lexicon.