Incident Management: The Complete Process Guide

An incident is an unplanned event that degrades or interrupts a service, or that threatens to. A web application returns errors to a share of its users. A payment provider stops accepting calls. A database fills its disk. A mistaken permission exposes data. What these have in common is that normal work stops being the priority and a response begins.

Incident management is the set of practices that make that response orderly. It answers a short list of questions in advance, while everyone is calm: how do we know something is wrong, who decides how serious it is, who leads, who is told, how do we record what we do, how do we know when it is over, and what do we change afterwards.

Teams that skip this preparation do not avoid the questions. They answer them during the incident, under pressure, in a chat thread, differently each time. This guide describes the stages of a mature process and what to put in place before the next incident.

Incident, problem and change: three different things

IT service management separates three ideas that are easily confused. An incident is an interruption that needs restoring now. A problem is the underlying cause of one or more incidents, investigated to prevent recurrence. A change is a deliberate modification to a system. A bad change often causes an incident, and an incident that keeps returning points to a problem.

The distinction matters in practice. During an incident the goal is to restore service, not to find the cause. Finding the cause belongs to the review and to problem management. Teams that try to diagnose fully before restoring service extend the outage.

Stage 1: Detection

An incident begins when the problem starts, not when someone notices it. The gap between the two is the time to detect, and it is the part of the response that monitoring controls. Good detection comes from alerts tied to symptoms that customers feel, such as error rates, latency and failed key journeys, rather than only from internal resource measures. Customers and colleagues are also sensors. Make it easy for them to report a problem and make sure the report reaches someone who can act.

Alerts that fire too often are ignored, and alerts that never fire are worthless. Review alerts after each incident and ask whether the right one fired, early enough, to the right person.

In Atlas, signed inbound webhooks from your monitoring can declare and update incidents automatically, and, if you switch that on, resolve them. A person can also declare one in a single step with a title and a severity. A double tap, or a retry on a poor connection, declares one incident rather than two, so a single outage does not page two sets of responders.

Stage 2: Triage and declaration

Declare early. A declared incident that turns out to be minor costs a few minutes. An undeclared incident that turns out to be serious costs trust. The first responder should be able to declare without asking permission, and should describe the symptom, not a suspected cause: "checkout returns an error for some users" rather than "database overloaded." Early theories are often wrong and anchor the response on the wrong thing.

At declaration, assign a severity. Severity answers how seriously to treat this and who to wake. If unsure, choose the higher level, because standing down is cheap and escalating late is not. The guide to incident severity levels explains how to define the scale.

Atlas keeps severity and impact as separate axes. Severity drives who is paged. Impact, which is none, degraded, partial outage or full outage, drives what customers are told. Both are recorded so that a loud but harmless alert and a quiet but damaging failure are treated differently.

Stage 3: Organise the response

The common failure in a serious incident is that everybody investigates and nobody coordinates. The remedy comes from emergency services, adapted by software teams as the Incident Command System: separate the people who fix from the person who leads and the person who communicates.

• Incident commander: owns the response, makes decisions, assigns work and does not usually investigate. The incident commander guide covers the role.
• Operations lead: directs the technical investigation and the fix.
• Communications lead: keeps stakeholders and customers informed on a schedule.
• Scribe: keeps the timeline, recording what was tried, what was found and when.
• Subject matter experts: join as needed for specific systems.
• Liaison: coordinates with other teams or outside parties.

Stage 4: Investigate and mitigate

Work to reduce customer impact first. The fastest mitigation is often not the fix. Rolling back a recent change, turning off a feature, shifting traffic or restarting a failing component restores service while the cause is still unknown. Ask early what changed recently, because the most common cause of an incident is a change. A release record that lists scope and deployments by environment answers that quickly.

Keep a running timeline. Post updates as you learn things, written for a reader arriving late and for the review that follows. Each update should state what is known, what is being tried and when the next update will come.

Atlas treats mitigated and resolved as different statuses on purpose. Mitigated means customer impact has ended, which is the number customers feel. Resolved means all work, including cleanup, is complete. Keeping them apart makes recovery time meaningful, and the incident statuses move through triage, investigating, identified, mitigated and resolved.

Stage 5: Communicate

Silence is read as incompetence or indifference. People inside the organisation need to know what is affected and when they will hear next. Customers need to know that you know, what they should do, if anything, and when you will update them. The communications guide in this series gives templates and a cadence.

Atlas does not provide a hosted status page for your own services. Customer communication in Atlas is by email update to affected customers, with an approval step. The subject and body that were approved are recorded together as a fingerprint that is checked again at the moment of sending, so editing an approved update revokes its approval. Sending to an empty recipient list is refused, and two people pressing send at once results in one message. Each sent update shows what happened to each customer's email: waiting, sent, delivered, bounced, marked as spam or not delivered.

Stage 6: Resolve and close properly

An incident is over when customer impact has ended, the cause is contained and the cleanup is done. Record how it ended, because the resolution type affects how you measure the response. A real fault that was fixed counts. A false alarm, a duplicate of another incident and expected behaviour do not, and Atlas excludes them from every response figure, because a quickly dismissed false alarm flatters the average and a slowly dismissed one damages it, and neither describes how well anyone handled a real problem.

Sometimes the problem returns. In Atlas, a resolved incident cannot change by a plain edit. An owner, an administrator or the incident commander can reopen it with a written reason of up to 500 characters. It returns to investigating, its resolution is cleared so that it counts as open, and the reopening is recorded on the timeline and the audit trail. A cancelled incident cannot be reopened. If it is a different problem, declare a new incident.

Stage 7: Learn

The review is where an incident pays for itself. A good review is blameless: it asks what conditions made the failure possible and what would have made it easier to detect or fix. It records several contributing factors rather than one root cause, because complex systems fail through interacting conditions, and naming a single cause usually means naming a person. Atlas does not have a root cause field. Publishing a review is refused below two contributing factors, and refused when no action item has an owner.

Action items should become real work. A follow-up that lives in a document nobody reopens is the reason incidents repeat. In Atlas each action item can be filed as a task in a project, assigned to its owner with a due date, and dropping an item requires a reason. The Incident reviews screen gathers overdue actions from every review in one list. When the same failure returns, record it as a known cause with a workaround so the next occurrence does not start from nothing. See the post-incident review structure guide.

Measure the response

Four intervals describe most of a response: time to detect, time to acknowledge, time to mitigate and time to resolve. Atlas calculates each from the recorded instants. Only the first acknowledgement counts, because if a later responder could overwrite it, the figure would improve every time someone else opened the incident. The metrics guide explains the definitions and their limits. Do not use averages as targets for individuals. They are a signal about the system.

Practice before you need it

People do not rise to the occasion in an incident. They fall back on what they have practised. The most effective preparation is a regular exercise in which the team handles a simulated incident, sometimes called a game day. A facilitator describes a symptom, answers questions about what the responders see and introduces complications, while the team follows its own process: declaring, assigning roles, communicating and deciding.

Keep the exercises short, about an hour, and focus on the process rather than the technology. Rotate the commander role so that more people have held it. Afterwards, hold a brief review as you would for a real incident, and fix what the exercise exposed: a missing contact, an out-of-date runbook, an escalation that pointed at someone who had left. Run a larger exercise, with the communication to a stand-in customer audience, once or twice a year.

What to prepare before the next incident

• A severity scale with response expectations for each level.
• An on-call schedule with an escalation path, so that an unacknowledged page reaches the next person. See the on-call management guide.
• Named roles and a deputy for each.
• A place to coordinate and a place to record the timeline.
• Communication templates and a rule about who can send them.
• Runbooks for your most likely failures. See the guide on runbooks and playbooks.
• A review habit with an owner for the action items.

Running incident management in Atlas

Atlas keeps the incident, the on-call rota, the timeline, the customer updates and the review in one place. Responders attach the Slack thread, email thread, Teams message, meeting recording or document by pasting its address, and Atlas recognises the source, removes tracking parameters and records it once. Each incident has a readable reference such as INC-1042, with a configurable prefix and format, safe to say aloud on a call. The whole incident can be downloaded as a report in PDF, Excel, Word or Markdown, all rendered from one source so that the formats cannot disagree.

Severity levels are yours to define, starting from presets and editing the levels, their response targets and whether each pages. Each incident stores the rank it was declared at, so renaming or retiring a level never rewrites history. The Atlas phone app lists incidents, open ones first, and lets you acknowledge, change status, resolve or reopen where your role allows.

The Atlas MCP server lets an assistant list and read incidents, read the timeline, post updates and declare one. Resolving, reopening and sending messages to customers are deliberately left out, because declaring early is safe and changing how an incident ended is not.

Incident Management, with on-call and incident reviews, is a module included from the Team plan. The pricing page lists plan details.

Common mistakes

• Treating the first responder as the commander for the whole incident, so that they investigate and coordinate at once.
• Debating the cause while customers are still affected.
• Updating stakeholders only when something has changed. Update on a schedule, even to say nothing has changed.
• Closing the incident when the pressure ends, not when the cleanup is done.
• Writing a review that names a person.
• Filing follow-ups that nobody owns.