Dears, I am trying to make a batch correction for scRNA dataset that has tumour and immune cells GSE162631, according to this paper: https://www.nature.com/articles/s41592-021-01336-8. I used Scanorama, but the results that I optained from the local machine is totally different from the remote machine. I know that this is a popular problem for the non-deterministic behaviour, but I used the same Docker image in both machines and include the following:
ENV OPENBLAS_NUM_THREADS=1
ENV MKL_NUM_THREADS=1
ENV OMP_NUM_THREADS=1
ENV VECLIB_MAXIMUM_THREADS=1
ENV NUMEXPR_NUM_THREADS=1
ENV NUMBA_NUM_THREADS=1
ENV BLIS_NUM_THREADS=1
ENV MKL_CBWR=COMPATIBLE
ENV MKL_DYNAMIC=FALSE
ENV OPENBLAS_CORETYPE=SANDYBRIDGE
ENV OPENBLAS_VERBOSE=1
in the Docker file to make sure that the results are the same on the 2 machines. However, the results still not the same like the attached figure:
I tried to use other tools like Harmony, the results are similar, but I have other datasets with complex batch effects, for which Scanorama is preferred to be used.
I really appreciate any help you can provide.
1 answer
but the results that I optained from the local machine is totally different from the remote machine.
The difference you show may not be biologically relevant as far as the results go. Have you looked into the results instead of just comparing the summary stats.
Here is a google gemini generated summary of why the results of running a docker tool on two CPU architectures may be different.
Docker tool results may differ between AMD and Intel CPUs due to variations in instruction sets, compiler optimizations, and architecture-specific behavior (e.g., AVX-512 vs. AVX2). Even with the same base linux/amd64 image, underlying hardware differences can lead to different timing, performance, or even numerical precision in compiled code.
Common Reasons for Discrepancies:
- Compiler Optimizations: Programs compiled with
-march=nativeor auto-vectorization can optimize differently for Intel's architecture versus AMD's Zen architecture, causing divergent results in mathematical or data-intensive workloads. - Instruction Set Utilization: One CPU might utilize specific instruction sets (like AVX-512) that the other does not support or executes differently, leading to varying speed and, rarely, output differences.
- Cache and Interconnect: Differences in cache hierarchies (CCX on AMD) or memory latency can cause different execution timing, which can alter outcomes in race-condition-prone software or performance benchmarks.
How to Ensure Consistent Results:
- Use Specific Flags: Avoid
-march=native. Explicitly define target architectures in your Dockerfile (e.g.,-march=x86-64-v3) for compiler flags to ensure identical machine code generation. - Verify Image Version: Ensure the exact same image hash (
image@sha256:...) is being pulled on both machines. - Control Environment Variables: Ensure that library behavior (e.g., MKL, OpenBLAS) is pinned to behave consistently, as they often detect CPU types and change algorithms.
While Docker provides environmental parity, it does not normalize the underlying physical CPU hardware capabilities.
Log in to answer this question.
So it's UMAP and clustering, both known to be non-deterministic. Did you set fixed seeds before running these functions?
yes I used a seed in both
I had a similar issue few years ago, related to processors differences and how they are handling digits
Consistency of Seurat SCTransform across computers/environments
https://github.com/satijalab/sctransform/issues/68