Identical PyTorch training runs give different results: seeds, cuDNN and CUBLAS_WORKSPACE_CONFIG

We compare model changes by training baseline and candidate with the same seed and data, then looking at validation metrics. A colleague noticed that two runs of the unchanged baseline, same commit and seed, ended at 87.42 and 87.11 percent accuracy. Our typical "improvement" threshold was 0.2 points, so noise alone could make a useless change look good.

Reproducing it

Three runs of the same config:

run   seed   val_acc   loss@step1000
a     42     87.42     0.6131
b     42     87.11     0.6127
c     42     87.29     0.6134

Loss diverged slightly from the first hundred steps. We turned on PyTorch's deterministic mode to find the cause, and training crashed on the first linear layer:

RuntimeError: Deterministic behavior was enabled with either `torch.use_deterministic_algorithms(True)`
or `at::Context::setDeterministicAlgorithms(true)`, but this operation is not deterministic because it
uses CuBLAS and you have CUDA >= 10.2. To enable deterministic behavior in this case, you must set an
environment variable before running your PyTorch application: CUBLAS_WORKSPACE_CONFIG=:4096:8 or
CUBLAS_WORKSPACE_CONFIG=:16:8.

What we ruled out

• Data order. We logged the first 100 sample ids per epoch; they matched across runs.
• Hardware differences. All runs used the same GPU model and driver; we also repeated on a single node.

Root cause

We seeded Python's random and torch, but several sources of nondeterminism remained:

1. torch.backends.cudnn.benchmark = True in our training script. cuDNN benchmarks several convolution algorithms at startup and picks the fastest, which can differ between runs, and some algorithms are not deterministic.
2. Nondeterministic CUDA kernels, notably cuBLAS operations that use workspace buffers in a way that can reorder floating point accumulation, and scatter-style reductions with atomic adds in our embedding code.
3. DataLoader workers seeded from the base seed without per-worker seeding of NumPy, so random augmentations in NumPy differed between runs.

What is happening under the hood

Floating point addition is not associative, so the order of summation changes results in the last bits. GPU kernels that use atomic operations or choose algorithms at runtime can sum in different orders on each run. Small differences early in training compound through millions of updates.

PyTorch provides controls:

• torch.manual_seed, plus seeding NumPy and Python RNGs.
• torch.backends.cudnn.benchmark = False and torch.backends.cudnn.deterministic = True.
• torch.use_deterministic_algorithms(True), which makes PyTorch use deterministic implementations where they exist and raise an error where they do not.
• CUBLAS_WORKSPACE_CONFIG, an environment variable that must be set before CUDA initialises, to make cuBLAS deterministic.

DataLoader workers each have their own RNG state. PyTorch seeds each worker's torch RNG differently based on the base seed, but NumPy and Python RNGs in workers need a worker_init_fn, and the base seed comes from a generator you pass to the DataLoader.

Environment

Minimal reproduction

torch.manual_seed(0)
torch.backends.cudnn.benchmark = True
# train for 1000 steps twice; compare losses -> small differences

The fix

A single function called at the very start of training, before anything touches CUDA:

import os, random
import numpy as np
import torch

def make_deterministic(seed: int) -> torch.Generator:
    os.environ.setdefault("CUBLAS_WORKSPACE_CONFIG", ":4096:8")   # must precede CUDA init
    random.seed(seed)
    np.random.seed(seed)
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True
    torch.use_deterministic_algorithms(True, warn_only=False)
    g = torch.Generator()
    g.manual_seed(seed)
    return g

def seed_worker(worker_id: int) -> None:
    worker_seed = torch.initial_seed() % 2**32
    np.random.seed(worker_seed)
    random.seed(worker_seed)

g = make_deterministic(cfg.seed)
loader = DataLoader(train_ds, batch_size=256, shuffle=True, num_workers=8,
                    worker_init_fn=seed_worker, generator=g)

We also set CUBLAS_WORKSPACE_CONFIG in the job's environment, because setting it from Python only works if no earlier import initialised CUDA.

One scatter-style reduction in our embedding code raised an error saying it does not have a deterministic implementation on CUDA. We replaced it with an equivalent formulation (sort by index, then a cumulative sum over segments) built from operations that do. The PyTorch documentation for use_deterministic_algorithms lists which operations are affected in each version.

Alternatives and trade-offs

• Determinism only for comparisons. Deterministic mode cost us about 9 percent in training speed. We enable it for evaluation runs that compare changes and for debugging, and use fast mode for production training.
• Multiple seeds instead of determinism. Train each variant with 3 to 5 seeds and compare means with confidence intervals. More compute, and it measures real seed sensitivity, which determinism hides. We do both for important decisions.

How to test the fix

Two runs of 2,000 steps in deterministic mode produced bit-identical losses at every step:

step   run_a        run_b
1000   0.61293812   0.61293812
2000   0.48810457   0.48810457

A CI job trains a tiny model twice for 200 steps and compares checkpoints with torch.equal on every parameter.

Confirming in production

• Repeat runs of the baseline in deterministic mode give identical metrics.
• With five-seed comparisons, we measured the real seed spread of our model at 0.25 points standard deviation, and updated the improvement threshold accordingly.

Regression guard

• The CI reproducibility check.
• Training configs record the seed, determinism flag and library versions in the run metadata.
• Code review flags new uses of cudnn.benchmark = True in comparison pipelines.

Security implications

None. Reproducibility does help audits: being able to retrain a model and get the same result is valuable when someone asks how a model in production was produced.

Prevention checklist

• Seed Python, NumPy and torch, and seed DataLoader workers with worker_init_fn and a generator.
• Disable cuDNN benchmark and enable deterministic algorithms when comparing runs.
• Set CUBLAS_WORKSPACE_CONFIG in the environment before CUDA initialises.
• Replace operations without deterministic implementations or accept documented nondeterminism.
• Compare important changes across several seeds.

References

• PyTorch reproducibility notes — https://pytorch.org/docs/stable/notes/randomness.html
• torch.use_deterministic_algorithms — https://pytorch.org/docs/stable/generated/torch.use_deterministic_algorithms.html
• cuBLAS reproducibility — https://docs.nvidia.com/cuda/cublas/index.html#results-reproducibility
• DataLoader randomness in multi-process loading — https://pytorch.org/docs/stable/data.html#randomness-in-multi-process-data-loading