Fine-tuned LoRA adapter produces garbage in production: loaded on the wrong base model revision

We fine-tuned a LoRA adapter on a 7B open model to summarise support tickets in our house style. Offline evaluation was strong. Two weeks after it went live, reviewers flagged summaries like:

Summary: The customer the customer reports reports that the the invoice invoice ...

Not every summary, but about a third, and always worse on tickets with product names.

Reproducing it

The training run's metadata recorded the base model and revision:

base_model: example-org/base-7b-instruct
base_revision: 3c1f9e2  (commit on the model hub, 2026-01-18)

The serving code:

base = AutoModelForCausalLM.from_pretrained("example-org/base-7b-instruct", torch_dtype=torch.bfloat16)
tok = AutoTokenizer.from_pretrained("example-org/base-7b-instruct")
model = PeftModel.from_pretrained(base, "s3-mounted/adapters/ticket-summary-v3")

No revision. The serving pods had restarted after a node rotation and downloaded the latest commit of the base repository, published on 2026-03-02, which the model's maintainers described as "updated tokenizer with expanded vocabulary and refreshed weights".

Loading the adapter on the pinned revision in a notebook reproduced good summaries; on the latest revision, broken ones.

What we ruled out

• Adapter corruption. Adapter files' hashes matched the training output.
• Generation settings. Same temperature and repetition settings in both cases.

Root cause

A LoRA adapter stores low-rank weight updates for specific layers of a specific base model. It assumes the exact base weights it was trained on, and the exact tokenizer, because token ids map to embedding rows. The new base revision changed both: different weights shifted what the adapter's deltas meant, and an expanded vocabulary changed tokenisation of many product names. Loading still "worked" because the shapes for the adapted layers were compatible, so nothing raised an error.

What is happening under the hood

PEFT saves an adapter_config.json with base_model_name_or_path, and optionally a revision. PeftModel.from_pretrained loads adapter weights onto whatever base model object you pass in; it does not verify that the base is the one used in training. On a model hub, a repository name without a revision resolves to the latest commit on the default branch, which can change at any time.

If the vocabulary size changed, you sometimes get a shape error for embedding layers (if they were trained or resized). If only weights or tokenizer merges changed, nothing fails, and quality degrades silently.

Environment

Minimal reproduction

Train an adapter on revision A of a base model, then load it with from_pretrained(name) after the repository has a new commit B with changed weights or tokenizer. Outputs degrade without an error.

The fix

1. Pin the base revision everywhere. Training writes the revision into the adapter config, and serving reads it from there:

import json

def load_with_adapter(adapter_dir: str):
    cfg = json.load(open(f"{adapter_dir}/adapter_config.json"))
    base_name, base_rev = cfg["base_model_name_or_path"], cfg["revision"]
    if not base_rev:
        raise RuntimeError("adapter has no pinned base revision")
    tok = AutoTokenizer.from_pretrained(base_name, revision=base_rev)
    base = AutoModelForCausalLM.from_pretrained(base_name, revision=base_rev, torch_dtype=torch.bfloat16)
    return PeftModel.from_pretrained(base, adapter_dir), tok

At training time:

peft_config = LoraConfig(r=16, lora_alpha=32, target_modules=TARGETS, task_type="CAUSAL_LM",
                         revision=BASE_REVISION)

2. Mirror base models internally. We copy each approved base revision into our own object storage and load from there. Upstream changes can no longer reach production without a deliberate step, and pods do not depend on the public hub being available.

3. Smoke evaluation at load time. Before a serving pod reports ready, it summarises five fixed tickets and checks simple properties (no repeated trigrams above a threshold, required fields present, length within bounds). A failing smoke test keeps the pod unready.

4. Save the tokenizer with the adapter. The training pipeline saves the tokenizer files next to the adapter, and serving loads the tokenizer from the adapter directory, so tokenizer and adapter always match.

Alternatives and trade-offs

• Merge the adapter into the base weights and ship a full model. No mismatch possible at load time; larger artifacts and less flexibility to swap adapters.
• Retrain on the new base. Sometimes worthwhile to get base model improvements; it must be a deliberate project with evaluation, not an accident.

How to test the fix

A CI test loads the adapter through the serving loader against the mirrored base revision and runs the smoke evaluation. A negative test deliberately loads a different revision and asserts the smoke test fails.

Confirming in production

• After redeploying with the pinned revision, reviewer flags returned to the pre-incident baseline of about 2 percent.
• All serving pods log base name, base revision, adapter version and tokenizer hash at startup.

Regression guard

• The loader refuses adapters without a pinned revision.
• Base models load only from our internal mirror.
• The load-time smoke evaluation gates readiness.

Security implications

Loading models by floating name from a public hub means anyone with write access to that repository, including an attacker who compromises it, can change what runs in your production. Pinning revisions, mirroring internally, and loading only safetensors files (no pickled code) close that supply chain gap. Model repositories deserve the same version pinning as software dependencies.

Prevention checklist

• Record and pin the base model revision with every adapter.
• Save and load the tokenizer with the adapter.
• Mirror approved base models internally.
• Run a smoke evaluation before a model pod becomes ready.
• Treat base model updates as deliberate upgrades with retraining and evaluation.

References

• PEFT LoRA configuration — https://huggingface.co/docs/peft/package_reference/lora
• PEFT model loading — https://huggingface.co/docs/peft/package_reference/peft_model
• Transformers from_pretrained revision argument — https://huggingface.co/docs/transformers/main_classes/model#transformers.PreTrainedModel.from_pretrained
• Hugging Face Hub, downloading specific revisions — https://huggingface.co/docs/huggingface_hub/guides/download