Writing blameless postmortems that actually change how your systems fail

Two years ago our payments team had the same incident three times in five months: a configuration flag flipped during a deploy, card authorisations started failing for one region, and someone rolled back after twenty to forty minutes. Each time we wrote a postmortem. Each time the conclusion was some version of "the engineer should have double-checked the flag", and each time the action item was "be more careful". The fourth occurrence finally made us read the three documents side by side. They were not postmortems. They were apologies with timestamps.

This guide is what we changed. It is aimed at engineers and leads who write or run postmortems and want them to produce fixes rather than guilt. By the end you should be able to write a timeline people trust, describe impact in numbers, look for contributing factors instead of a person to blame, write action items that actually close, and run a review meeting that makes the next incident less likely.

What "blameless" means and what it does not

Blameless does not mean nobody is accountable, and it does not mean pretending mistakes did not happen. It means we assume that the people involved acted reasonably given what they knew, the tools they had and the pressure they were under, and that the useful question is why that reasonable action led to a bad outcome.

The practical reason is information. If writing "I ran the migration against the wrong cluster" costs someone their reputation, the next person will write "a migration was run" and leave out the detail that explains everything: both clusters had identical prompts in the terminal. Google's SRE book makes the same argument in its chapter on postmortem culture: a blamed person stops reporting, and an organisation that cannot see its near-misses cannot fix them.

A useful test while editing: replace every person's name with their role. If a sentence still reads as an accusation ("the on-call engineer ignored the alert"), rewrite it to describe the system ("the alert fired into a channel with 400 other alerts that day; it had been noisy for weeks").

When to write one

Decide your triggers in advance so nobody has to argue about it at 2 a.m. Ours are:

• Any customer-visible outage or degradation longer than 15 minutes.
• Any data loss or corruption, however small.
• Any security incident, including ones caught internally.
• Any incident that needed a manual rollback in production.
• Any near-miss that someone on the team thinks was lucky.

The last one produces some of our best documents. A near-miss has all the contributing factors of an outage and none of the stress, so people are more candid.

The structure we use

We keep a short template in the wiki. Every section exists because a past postmortem was weaker without it.

## Summary
Two or three sentences a non-engineer can read.

## Impact
Who was affected, for how long, and how badly. Numbers, not adjectives.

## Timeline
All times in UTC. Detection, escalation, key decisions, mitigation, resolution.

## Contributing factors
What made this possible, what made it worse, what made it last longer.

## What went well
Things we want to keep doing.

## Where we got lucky
Things that limited the damage but that we cannot rely on.

## Action items
Owner, ticket, priority, due date.

## Supporting material
Graphs, logs, chat excerpts, links to dashboards.

Summary

Write it last. It should say what broke, for whom, for how long, and the single most important thing we are changing. If a director reads only this paragraph, they should not get the wrong idea.

Impact

Adjectives like "significant" and "minor" hide disagreement. Write the numbers you have and say how you got them:

If you cannot measure impact, that is itself a finding and usually an action item.

Timeline

The timeline is the backbone. Build it from evidence, not memory: chat logs, deploy history, alert history, dashboards, audit logs. Use one timezone, preferably UTC, and include the moments where people made decisions, not only the moments where systems changed.

09:02  Deploy 2025-04-08.3 starts (feature flag config bundled)
09:14  EU auth error rate rises from 0.1% to 4%; no alert (threshold 5%)
09:21  Support ticket mentions login failures in Germany
09:29  Support escalates to on-call in incident channel
09:33  On-call checks dashboard, suspects identity provider; opens vendor status page (green)
09:41  Second engineer notices deploy at 09:02, proposes rollback
09:44  Rollback starts
09:51  Error rate back to baseline

Two gaps jump out of a timeline like this: 15 minutes from impact to anyone knowing (the alert threshold), and 12 minutes spent on the wrong hypothesis. Those gaps are where the action items come from.

Contributing factors, not root cause

Most real incidents have several causes that only hurt in combination. Writing a single "root cause" pushes the author to pick one, and the one picked is often the last human action, because it is the most visible. We list contributing factors in three groups:

• What made it possible: the config flag and the code shipped in the same deploy; nothing validated the flag value per region.
• What made it worse: retries from the mobile client tripled load on the auth service during the incident.
• What made it last longer: no alert below 5% errors; deploy events were not shown on the auth dashboard, so nobody connected the timing.

Ask "how" and "what" rather than "why" when interviewing people. "What did the dashboard show you at 09:33?" gets a description. "Why didn't you roll back sooner?" gets a defence.

What went well and where we got lucky

"What went well" protects the good habits from being optimised away. "Where we got lucky" is the most underrated section in the template. In the incident above, the deploy happened at 09:02 UTC, when US traffic was near its daily low. If it had shipped six hours later, the same bug would have hit three times as many people. Luck is a risk you have not mitigated yet.

Action items that get done

Action items are where most postmortems quietly fail. Ours have four rules:

1. Every item has one owner and a ticket. Not a team, a person. The ticket lives in the owning team's backlog, not in the postmortem.
2. Every item is specific and checkable. "Improve monitoring" cannot be closed. "Alert when EU auth error rate exceeds 1% for 3 minutes" can.
3. Every item has a priority tied to risk. We tag items as prevent (stops recurrence), detect (finds it faster), or mitigate (reduces blast radius). A postmortem with only detect items is a warning sign.
4. No "be more careful". Humans will keep making the same mistakes under the same conditions. Change the conditions: a guardrail, a check, a default, a tool.

We also prefer a small number of items. Five items that ship beat fifteen that rot. If a fix is large, the action item is to write the design doc, with a date.

Running the review meeting

The document is drafted by the incident commander or someone close to the response within five working days, while memories and logs are fresh. Then we hold a 45-minute review:

• Invite the responders, the owning teams and one outsider. The outsider asks the naive questions that insiders skip.
• The facilitator is not the author and not the most senior person in the room. Their job is to keep language about systems and to stop the conversation turning into a trial.
• Walk the timeline first. Correct it together before discussing causes. Disagreements about facts are easier to settle than disagreements about blame.
• Agree on owners in the meeting. Items without an owner by the end of the meeting tend to stay that way.

Publish the final document widely. A postmortem that only the involved team reads teaches one team.

Following up

Thirty days after the review, someone (we rotate it) checks every open action item and posts the status in the incident channel. Once a quarter we read all postmortems from the period together and look for patterns: the same service, the same type of change, the same missing alert. That quarterly read is where we spotted that config-bundled-with-code deploys were behind four incidents in a year, which led to a separate config pipeline with its own staged rollout. No single postmortem would have justified that project.

We also track two numbers: the percentage of action items closed within their due date, and the number of repeat incidents (same contributing factor within 6 months). The first went from about 35% to 80% after we introduced the four rules above. The second is the number we actually care about.

Common mistakes

Key takeaways

• Blameless means asking why a reasonable action led to a bad outcome, not pretending nobody acted.
• Impact and timeline must come from evidence and numbers; vague words hide disagreement.
• Look for several contributing factors, sorted by what enabled, worsened and prolonged the incident.
• Action items need an owner, a ticket and a checkable result, and should change conditions rather than ask people to try harder.
• The review meeting and the 30-day follow-up matter as much as the document.
• Read postmortems in batches; patterns across incidents justify fixes that single incidents cannot.

References

• Google SRE Book, Postmortem Culture: Learning from Failure — https://sre.google/sre-book/postmortem-culture/
• Google SRE Workbook, Postmortem Culture — https://sre.google/workbook/postmortem-culture/
• PagerDuty Postmortem Documentation — https://postmortems.pagerduty.com/
• Atlassian Incident Management Handbook, Blameless postmortems — https://www.atlassian.com/incident-management/postmortem/blameless