Batched LLM generation garbles short prompts: right-padding with a decoder-only model

We classify support tickets into topics with a small instruction-tuned model running locally through Hugging Face Transformers. Moving from one-at-a-time generation to batches of 16 made the job five times faster, and the quality report the next morning showed accuracy dropping from 91 to 54 percent.

Looking at outputs for one batch, the longest prompt got a sensible label and others got fragments:

prompt_len  output
412         "billing"
188         " the the the customer is is"
95          "\n\n\n\n<|eot_id|>"

The logs contained a warning we had not noticed:

A decoder-only architecture is being used, but right-padding was detected! For correct generation
results, please set `padding_side='left'` when initializing the tokenizer.

Reproducing it

tok = AutoTokenizer.from_pretrained(MODEL)
tok.pad_token = tok.eos_token
batch = tok(prompts, return_tensors="pt", padding=True)
out = model.generate(**batch, max_new_tokens=8)

With a single prompt, output was correct. With prompts of different lengths, only the longest one was.

What we ruled out

• Sampling randomness. We used greedy decoding.
• Prompt formatting. The chat template output was identical in single and batch mode.
• Model or dtype issues. Single-prompt runs in the same process were fine.

Root cause

The tokenizer padded on the right by default. For a decoder-only model, generation continues from the last position of each sequence. With right padding, the last positions of shorter sequences are pad tokens, so the model generates as if the prompt ended with several padding tokens (here the end-of-sequence token, since we had set pad equal to EOS). The outputs for shorter prompts were continuations of padding, not of the prompt.

What is happening under the hood

Batched tensors need equal lengths, so shorter sequences get pad tokens. Where padding goes matters:

• Right padding: [prompt tokens][pad pad pad]. Fine for encoder models and training with masks, broken for generation, because new tokens are appended after the pads.
• Left padding: [pad pad pad][prompt tokens]. All prompts end at the same position, so new tokens follow the real text.

The attention mask tells the model to ignore pad positions; with left padding, position ids are also computed from the mask so real tokens get correct positions. Many decoder-only tokenizers have no pad token, so people set it to EOS, which works only with a correct attention mask and left padding.

Environment

Minimal reproduction

prompts = ["Classify: refund not received for order 1234. Topic:", "Classify: login fails. Topic:"]
tok.padding_side = "right"
print(tok.batch_decode(model.generate(**tok(prompts, return_tensors="pt", padding=True), max_new_tokens=5)))

The fix

from transformers import AutoTokenizer, AutoModelForCausalLM

tok = AutoTokenizer.from_pretrained(MODEL, padding_side="left")
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype="bfloat16", device_map="cuda")
model.generation_config.pad_token_id = tok.pad_token_id

batch = tok(prompts, return_tensors="pt", padding=True).to(model.device)
out = model.generate(
    input_ids=batch["input_ids"],
    attention_mask=batch["attention_mask"],      # always pass the mask
    max_new_tokens=8,
    do_sample=False,
)
new_tokens = out[:, batch["input_ids"].shape[1]:]  # strip the (left-padded) prompt
labels = tok.batch_decode(new_tokens, skip_special_tokens=True)

Two details: slicing off the prompt uses the padded input length, which is the same for all rows with left padding; and the training code for this model keeps right padding, so we set padding_side explicitly in each place instead of relying on a shared tokenizer object.

We also sort prompts by length before batching. Similar lengths mean less padding and faster batches.

Alternatives and trade-offs

• A serving engine like vLLM handles batching and padding internally with continuous batching, and is much faster for large volumes. We moved to it later; this fix kept the existing job correct in the meantime.
• Batch size 1. Correct and slow.

How to test the fix

A unit test generates for a batch of prompts with very different lengths and compares each output with single-prompt generation, which must be identical under greedy decoding:

def test_batched_matches_single(model, tok):
    prompts = [SHORT, MEDIUM, LONG]
    batched = generate_batch(prompts)
    single = [generate_batch([p])[0] for p in prompts]
    assert batched == single

Confirming in production

• Accuracy on the daily labelled sample returned to 91 percent.
• Throughput stayed five times higher than single-prompt generation; sorting by length added another 20 percent.
• The right-padding warning no longer appears in logs.

Regression guard

• The batched-versus-single test in CI.

• Warnings from transformers are captured and reviewed; this specific warning is turned into an error in the job with warnings.filterwarnings("error", message=".*right-padding was detected.*").

• The batching helper used by all our offline generation jobs now lives in a shared package with left padding hardcoded for generation, so individual jobs cannot pick the wrong side by accident.

• Job reports include a per-batch comparison of output lengths; a sudden drop in average output length is often the first visible sign of a padding or template problem.

Security implications

None directly. The job processes support ticket text with personal data, which is why it runs on our own GPU instead of an external API, and outputs are stored without the ticket text.

Prevention checklist

• Use left padding for batched generation with decoder-only models.
• Always pass the attention mask to generate.
• Set a pad token explicitly and configure pad_token_id.
• Test that batched output equals single-prompt output under greedy decoding.
• Treat library warnings in ML jobs as potential correctness bugs.

References

• Hugging Face, LLM tutorial: padding side for generation — https://huggingface.co/docs/transformers/main/en/llm_tutorial#wrong-padding-side
• Tokenizer padding and truncation — https://huggingface.co/docs/transformers/main/en/pad_truncation
• GenerationConfig — https://huggingface.co/docs/transformers/main/en/main_classes/text_generation