Post-Incident Review: Structure, Questions and Follow-Up

A post-incident review, also called a postmortem, is a written account of an incident and, more importantly, of what will change as a result. For the principles of running the meeting itself, see the guide to running a blameless postmortem. This guide concentrates on the structure of the document and on follow-up.

Decide which incidents get a review

Set a threshold in advance, for example every incident at or above a given severity, any incident with customer data involved, any that required a rollback, and any that surprised the team. Decide too which ones do not: a false alarm or a duplicate of another incident does not need its own review.

Hold the review soon after resolution, while memory is fresh, but not in the same hour as the incident. A day or two is typical.

The structure

• Summary: three or four sentences a busy reader can understand, stating what happened, who was affected and for how long.
• Impact: what customers and the business experienced, with figures where available.
• How it was noticed: the detection route and the time between start and detection.
• Timeline: the key events in order, each with a time. The scribe's record is the source.
• What was done: the actions that helped and the ones that did not.
• Contributing factors: the conditions that allowed the incident and delayed recovery.
• What went well: practices worth keeping. Learning from success matters as much as learning from failure.
• What was learned: surprises and gaps in understanding.
• Action items: each with an owner and a due date.

Contributing factors, not a root cause

Complex systems rarely fail for one reason. A change, a latent defect, a missing alert, an untested assumption and a tired responder may all have played a part. Asking for "the root cause" pushes people to choose one, and the easy choice is the last person who touched the system. Asking for contributing factors invites a more accurate picture.

Atlas does not provide a root cause field. The review records several contributing factors, and publishing is refused below two. Useful questions are: what made the mistake easy to make, what would have made it visible sooner, and what made recovery slower than it needed to be.

Write actions that get done

• Specific. "Add an alert on queue depth above a stated threshold" is an action. "Improve monitoring" is not.
• Owned. Every action has one named owner. An item owned by everybody is owned by nobody. Atlas refuses to publish a review when no action item has an owner.
• Dated. Give each a due date that reflects its importance.
• Proportionate. Prefer a few actions that will be done to a long list that will not. Rank them by the risk they reduce.
• Tracked as work. Put the action in the same system as other work. In Atlas each action item can be filed as a task in a project, assigned to its owner and given its due date, and dropping an item requires a reason so a reader later can tell a decision from an omission.

Keep follow-up visible

Reviews fail not in the writing but in the follow-through. Report overdue actions regularly to the people who can unblock them. Atlas gathers overdue actions from every review in one Follow-up owed list, because an action overdue inside one review is invisible and the same action in a shared list is not. Treat a rising backlog of overdue actions as a finding about the process.

Recurring failures

When the same failure appears a second time, writing another review teaches little. Record the failure as a known cause, with a workaround, so that the next responder does not start from nothing. Atlas keeps these as known causes on the Incident reviews screen. The problem management guide explains how this fits the wider practice.

Share the review

Publish the review to the people who can learn from it. Reviews that remain in a private folder protect nobody. Remove names where they do not serve the learning, and keep the language factual.