Practical Guide: Running a Blameless Postmortem for a Production Incident

Most teams say they want blameless postmortems.

In practice, many postmortems quietly become one of these:

• a search for the person who "made the mistake",
• a timeline that avoids uncomfortable decisions,
• a document that gets written, shared once, and never used again.

A real blameless postmortem is different.

It treats the incident as a systems problem, not a character flaw. It aims to improve the system’s reliability by:

• creating a shared, accurate understanding of what happened,
• surfacing how the system and the organisation shaped the outcome,
• producing follow-ups that actually reduce future risk.

This guide is a practical approach you can run in production teams. It is opinionated, but it is designed for reality: limited time, mixed seniority, and the pressure to "move on".

What "Blameless" Actually Means (and What It Doesn’t)

"Blameless" is often misunderstood.

It does not mean:

• nobody is accountable,
• we pretend mistakes did not happen,
• we avoid hard conversations.

It means something more precise:

• We assume people acted reasonably based on what they knew at the time.
• We treat failures as a product of context: incentives, tooling, system design, and information.
• We focus on changing the system so the same class of failure is less likely.

Blamelessness is not softness. It is a reliability technique.

When to Run the Postmortem

Timing matters.

Checklist:

• Run it soon: within 24–72 hours while memory is fresh.
• Not too soon: ensure the incident is stabilised and people have slept.
• Time-box it: 45–90 minutes for most incidents.

If you wait weeks, the postmortem becomes politics. If you do it immediately, it becomes therapy or argument.

Who Should Be in the Room

A small group is better than a crowd.

Invite:

• the primary responders / on-call engineers,
• a representative from any team whose system contributed materially,
• someone who represents the user impact (support/product),
• a facilitator who is not the "main protagonist" of the incident.

Avoid turning it into a public trial. If you have a large org, you can share the write-up later.

Roles: Facilitator, Scribe, and Owner

Assign roles explicitly.

• Facilitator

  • keeps the conversation structured,
  • watches for blame language,
  • surfaces assumptions and missing context,
  • keeps the meeting time-boxed.

• Scribe

  • captures the timeline and key facts,
  • writes down contributing factors,
  • records action items with owners and due dates.

• Incident owner (optional)

  • owns follow-through after the meeting.

If you do not assign ownership, action items will decay into "someone should".

Step 1: Start With Psychological Safety Rules (2 minutes)

Open with simple ground rules.

Suggested script:

• "We are here to understand the system, not to judge individuals."
• "Assume everyone acted with good intent and partial information."
• "We will focus on what made the incident possible and what made it hard to detect or recover."

This sounds basic, but it changes the tone.

Step 2: Establish a Shared Timeline (15–20 minutes)

The timeline is the backbone.

Rules:

• Keep it factual and timestamped.
• Prefer data sources (alerts, logs, dashboards, tickets) over memory.
• Include when information became available, not just when things happened.

Template:

• T-?: Conditions existed (latent risk).
• T0: Trigger event.
• T+X: Detection.
• T+X: First mitigation attempt.
• T+X: Escalations.
• T+X: Recovery.
• T+X: Customer comms.

A useful question during the timeline:

• "What did we believe was happening at this point?"

This surfaces gaps between reality and perception.

Step 3: Identify Impact in Concrete Terms (5–10 minutes)

Be specific. "The site was down" is rarely the right level.

Capture:

• user-visible symptoms,
• affected cohorts (all users, a region, enterprise customers only),
• duration,
• data integrity implications,
• internal cost (pages, incident time, support load).

This helps later when you prioritise follow-ups.

Step 4: Ask "How Did This Make Sense at the Time?" (15–20 minutes)

This is the blameless core.

For key decision points, ask:

• What signals did responders have?
• What signals were missing or misleading?
• What constraints existed (time, access, tooling, approvals)?
• What was the fastest reasonable move under pressure?

You are trying to understand the local rationality of actions.

When you do this well, you naturally stop blaming. The system becomes visible.

Step 5: Separate Root Cause From Contributing Factors

Teams often try to compress an incident into one cause:

• "It was a bad deploy."
• "It was an AWS issue."
• "It was human error."

In reality, incidents typically require multiple conditions.

A helpful structure:

• Trigger: what started the incident.
• Vulnerability: what allowed the trigger to create impact.
• Detection gap: what made it hard to notice early.
• Recovery gap: what made it hard to mitigate quickly.

This avoids the false comfort of a single root cause and produces better action items.

Step 6: Generate Action Items That Reduce Risk (15–25 minutes)

Action items should be small, concrete, and measurable.

Good action items:

• reduce probability (prevent),
• reduce time-to-detect (detect),
• reduce time-to-recover (recover),
• reduce blast radius (contain).

Examples:

• Add a specific alert on a critical SLO that was missing.
• Add a circuit breaker / timeout to isolate a failing dependency.
• Automate a rollback step that was manual and error-prone.
• Add a runbook section for the failure mode you encountered.

Anti-patterns:

• "Be more careful."
• "Improve monitoring." (without specifying what and where)
• "Add more tests." (without identifying which risk)

Every action item needs:

• an owner,
• a due date,
• and a definition of done.

Step 7: Write the Postmortem in a Format People Will Actually Use

If your postmortems feel like homework, they will be ignored.

A pragmatic format:

• Summary (2–5 sentences)
• Impact (who/what/when)
• Timeline (timestamped)
• Contributing factors (grouped by trigger/vulnerability/detection/recovery)
• What went well (so you keep the good parts)
• What we will change (action items)
• Links (dashboards, logs, PRs, runbooks)

Keep it honest and readable.

Step 8: Follow-Through (the Part Most Teams Skip)

If action items do not land, postmortems become theatre.

Follow-through practices that work:

• Review postmortem actions weekly in the same forum as delivery planning.
• Track actions like real work, not "nice-to-have" tasks.
• Close the loop by updating runbooks and docs.
• Revisit the incident after 30–60 days: what changed, what didn’t?

Reliability improves when learning turns into system change.

Common Failure Modes (and How to Avoid Them)

A few patterns we see repeatedly.

1. Blame Disguised as "Accountability"

Language like "Who approved this?" can be valid, but only if you also ask:

• What incentives and pressures shaped the approval?
• What information was missing?
• What guardrails should exist?

2. The Timeline Becomes the Whole Postmortem

A timeline is necessary, not sufficient.

If you only produce a timeline, you do not have learning. You have a record.

3. Action Items Become a Wish List

Too many actions means none get done.

Prefer 3–7 high-leverage actions over 20 vague ones.

4. "Blameless" Becomes "No Standards"

Blameless does not mean you cannot improve engineering discipline.

It means you improve discipline by building better systems and guardrails, not by hoping people never slip.

How We Approach Postmortems at Fentrex

When we help teams with reliability, we look at incidents as architectural signals.

Incidents often reveal:

• hidden coupling between components,
• missing guardrails in deployment and change management,
• observability gaps,
• operational workflows that do not match the system’s complexity.

A good postmortem is one of the most cost-effective ways to surface these.

Questions to Ask Before Your Next Postmortem

• Are we optimizing for learning or for looking competent?
• Do we have enough data to build a real timeline?
• Did we capture why actions made sense at the time?
• Do our action items change the system, or only remind people?

If you can answer these honestly, you are much closer to a postmortem culture that actually improves reliability.