Whisper transcripts invent text during silence and repeat phrases

We transcribe customer calls for quality review using a self-hosted Whisper model. Reviewers started flagging transcripts with lines that were never spoken. Two patterns showed up again and again:

[00:41.20 -> 00:47.80]  Thank you for watching, and don't forget to subscribe.
[00:47.80 -> 00:52.10]  Thank you for watching, and don't forget to subscribe.
...
[03:12.00 -> 03:30.00]  I'll check the account and call you back. I'll check the account and call you back.

The first appeared in segments where the caller was on hold with music. The second was a real sentence from earlier in the call repeated during a long silence.

Reproducing it

We cut a 2-minute clip with 40 seconds of hold music in the middle and ran it ten times. Every run produced invented text in the music section. A clip of pure room noise produced a short "Thank you." on most runs.

What we ruled out

• Audio corruption. Waveforms and spectrograms of the hold sections showed music and silence, nothing that sounded like speech.
• Wrong language setting. We forced English; the problem remained.
• Model size. The large model hallucinated less often than the small one, but still did it.

Root cause

Whisper is a sequence-to-sequence model trained on a huge amount of subtitled audio from the web. It does not have a reliable "no speech" mode: when given audio without speech, it tends to produce the most likely text, which for web videos is often an outro like "thanks for watching". On top of that, by default each 30-second window is decoded with the previous window's text as a prompt (condition_on_previous_text=True). Once a phrase is produced, the next window is primed to repeat it, and silence gives the model nothing to break the loop.

What is happening under the hood

Whisper processes audio in 30-second windows. For each window it predicts a "no speech" probability and decodes text with a beam or greedy search. The reference implementation has a few heuristics to catch bad output:

• no_speech_threshold: if the no-speech probability is above this value and the average log probability is low, the segment is treated as silence.
• logprob_threshold: if the average log probability of the decoded text is below this, decoding is retried at a higher temperature or the segment is considered failed.
• compression_ratio_threshold: repeated text compresses very well with gzip; a high ratio signals a repetition loop and triggers a retry.

These help but do not solve the core problem: music and noise do not always look like "no speech" to the model. Removing non-speech audio before Whisper sees it is far more reliable than detecting bad output after.

Environment

Minimal reproduction

from faster_whisper import WhisperModel

model = WhisperModel("large-v3", device="cuda", compute_type="float16")
segments, _ = model.transcribe("call_with_hold_music.wav", language="en")
for s in segments:
    print(f"[{s.start:.2f} -> {s.end:.2f}] {s.text}")

The fix

1. Voice activity detection before decoding. faster-whisper ships with a Silero VAD integration that removes non-speech audio before decoding and maps timestamps back to the original audio.

segments, info = model.transcribe(
    "call.wav",
    language="en",
    vad_filter=True,
    vad_parameters={"min_silence_duration_ms": 700, "speech_pad_ms": 200},
    condition_on_previous_text=False,
    no_speech_threshold=0.6,
    log_prob_threshold=-1.0,
    compression_ratio_threshold=2.2,
    temperature=[0.0, 0.2, 0.4],
)

2. Do not condition on previous text. Setting condition_on_previous_text=False breaks the repetition loop at the cost of slightly less consistent spelling of names across windows. For calls, that trade-off was clearly worth it.

3. Post-filter what still slips through. A small set of rules drops segments that are almost certainly invented: known outro phrases in segments with low average log probability, and consecutive identical segments.

KNOWN_HALLUCINATIONS = {"thank you for watching", "thanks for watching", "please subscribe"}

def keep(segment, previous_text: str | None) -> bool:
    text = segment.text.strip().lower().rstrip(".!")
    if segment.avg_logprob < -0.8 and any(p in text for p in KNOWN_HALLUCINATIONS):
        return False
    if previous_text and text == previous_text:
        return False
    return True

Alternatives and trade-offs

• Stronger VAD settings (longer minimum silence, higher speech threshold) remove more hallucinations and risk cutting quiet speech, like a customer mumbling an account number.
• Separate music detection. For hold music specifically, a music classifier can mark sections. VAD handled most of it for us.
• Speaker diarisation first, then transcribe per speaker turn. Higher quality for multi-speaker audio and naturally skips silence; more compute and complexity.

How to test the fix

We built an evaluation set of 150 call clips with human transcripts, including 40 with hold music or long silences, and measured word error rate plus an "inserted words in non-speech regions" count:

config                     WER     inserted_words_in_nonspeech
baseline                   14.8%   1,912
+ vad_filter               12.9%     214
+ no condition_on_prev     12.6%      61
+ post-filter              12.6%      18

Confirming in production

• Reviewer flags for "text not spoken" dropped by about 95 percent in the first month.
• Average transcript length per call fell by 6 percent, mostly removed invented text.
• Processing time dropped 20 percent because VAD skips silent audio before decoding.

Regression guard

• The 150-clip eval runs on any change to model, library version or parameters.
• A daily job counts segments matching known hallucination phrases and alerts on spikes.
• Transcripts store the parameters used, so we can re-run affected calls after changes.

Security implications

Invented text in a transcript is an integrity problem with real consequences: quality reviews, disputes and compliance records rely on these transcripts. We added a notice in the review UI that transcripts are machine-generated and link each segment to its audio timestamp, so reviewers can check the source before acting. Transcripts also contain personal data and follow the same retention rules as the recordings.

Prevention checklist

• Run VAD before Whisper for any audio with silence, music or noise.
• Turn off conditioning on previous text for long, gappy audio.
• Use the decoding thresholds, and measure their effect.
• Keep an eval set with non-speech sections and count inserted words.
• Link transcript segments to audio for human verification.

References

• OpenAI Whisper repository and transcribe options — https://github.com/openai/whisper
• faster-whisper (VAD filter) — https://github.com/SYSTRAN/faster-whisper
• Silero VAD — https://github.com/snakers4/silero-vad
• Robust Speech Recognition via Large-Scale Weak Supervision (Whisper paper) — https://arxiv.org/abs/2212.04356