Problem Management: From Repeat Incidents to Known Errors

In IT service management, a problem is the underlying cause of one or more incidents. Problem management is the practice of finding that cause, deciding what to do about it and recording what is known in the meantime. It is distinct from incident management, which aims to restore service as quickly as possible, whatever the reason it broke.

The two compete for attention. During an incident the priority is to restore service, and the investigation of cause is secondary. After the incident, the pressure fades and the cause is rarely pursued. Problem management exists to make sure that it is.

Incident, problem and known error

• Incident: an unplanned interruption or degradation of a service. The goal is restoration.
• Problem: the cause of one or more incidents. The goal is understanding and prevention.
• Workaround: a way to reduce or avoid the impact of an incident or problem without removing the cause.
• Known error: a problem whose cause is understood and for which a workaround exists, but which has not been fixed. A known error is a live liability, not a closed matter, and so it is tracked separately from a resolved problem.

Two modes: reactive and proactive

Reactive problem management responds to incidents that have happened. After an incident, or a group of related incidents, someone investigates the cause. Proactive problem management looks for problems before they cause serious incidents: it reviews trends, near misses, alerts that fire repeatedly and the contributing factors recorded in past reviews.

Most teams start reactive, and that is right. Proactive work follows when the reactive process runs reliably.

Step 1: Detect and record the problem

A problem is raised when an incident reveals an unresolved cause, when the same symptom returns, or when analysis of past incidents shows a pattern. A useful rule: open a problem the second time you see the same failure. By then, writing another incident review teaches little, and what is needed is a record of the cause itself.

Give each problem a clear title, a description of the symptoms, a list of related incidents and an owner. The owner is accountable for driving it to a conclusion, even if others do the investigation.

Step 2: Investigate

Investigation is a disciplined search for the conditions that produced the incident. Gather evidence: timelines, logs, change records and the contributing factors from reviews. Look at what changed before the first occurrence, and what is common across the incidents. Test hypotheses rather than collecting opinions.

Several techniques help. A timeline ordered by time shows sequence and delays. Asking why repeatedly, five times or as many as needed, moves from symptom to condition, with the caution that it can force a single chain of reasons where there are several. A fishbone diagram groups possible contributors by category. Whatever the method, keep the blameless principle: the aim is to understand how the system allowed the failure.

Avoid a single root cause where the facts show interaction. Atlas reviews record contributing factors and do not have a root cause field, and publishing a review is refused below two factors.

Step 3: Find a workaround and record the known error

A permanent fix may take weeks or need a design change. Meanwhile the problem can recur. Document a workaround that responders can apply, and record the problem as a known error so that the next person finds it quickly. A known error entry should state the symptoms, the cause as far as it is understood, the workaround and the status of the permanent fix.

In Atlas, recurring failures are recorded as known causes on the Incident reviews screen, with a workaround, so that the next occurrence does not start from nothing. Treat that record as part of the on-call toolkit. The known error database guide describes how to keep it useful.

Step 4: Fix and verify

Decide what permanent fix is justified. Weigh the cost and risk of the fix against the cost of the recurring incidents. Sometimes the right decision is to accept the problem and keep the workaround, and that decision should be recorded with its reason and an owner, not left to drift.

Treat the fix as a change. Plan it, review it and release it with the same care as any other, including a rollback plan. See the guide to IT change management and the release management guide. After the fix ships, watch for recurrence over a period long enough to be confident, and only then close the problem.

Step 5: Close the loop

• Link the problem to the incidents it caused, so that the history is findable.
• Convert fix and prevention actions into tracked tasks with owners and dates. In Atlas an action item from a review can be filed as a task, and overdue actions are gathered in the Follow-up owed list across all reviews.
• Update the runbooks and alerts that the problem exposed.
• Share what was learned with the teams that could meet a similar problem.

Prioritising problems

Not every problem deserves a permanent fix. Teams that try to fix everything fix nothing. Rank problems by the harm they cause and the effort to fix them. Useful inputs are how often the related incidents occur, how severe they are, how many customers they affect, how much responder time each occurrence consumes and whether the problem carries a risk of data loss or breach.

A simple method is to estimate, for each problem, the cost of one more year of the status quo and the cost of the fix. Where the fix is cheap and the cost of waiting is high, do it now. Where the fix is expensive and the workaround is reliable, accept the problem formally, record the decision and set a date to look again. An accepted problem with an owner and a review date is managed. One that is simply ignored is a hidden liability.

A worked example

A service goes slow on the first working day of each month, and on call is paged for it three months running. Each incident was resolved by restarting a worker process, and each review recorded the restart as the fix. After the third occurrence, the team opens a problem.

The owner gathers the three timelines and notices that each began within an hour of the monthly billing export. A change record shows that the export was moved to run in the same process pool as customer requests four months ago, and that its memory use grows with the number of accounts. The contributing factors are listed: the export shares resources with live traffic, memory use is not alerted on, the change was not load tested against month-end volumes and the restart masks the symptom.

The team records a known error with the symptoms, the cause as understood and the restart as a workaround, and links it from the alert. It assigns three actions: move the export to a separate worker pool, add an alert on memory growth, and add a month-end volume test to the readiness checklist for changes to batch jobs. The first action ships in the next release, and the problem stays open as a known error until the first month-end after it has passed without incident. Then it is closed, and the known error entry is retired.

Proactive techniques

• Trend analysis: group incidents by service, symptom and contributing factor each quarter. A cluster that nobody called a problem is one.
• Alert review: the alerts that fire most often point to chronic conditions. See the alert fatigue guide.
• Near misses: record events that could have caused an incident. They are free lessons.
• Change review: look at the changes that caused incidents, and at what they had in common.
• Dependency review: list the single points of failure and the components nobody understands.

Roles

Name a problem owner for each problem, accountable for driving it to a decision. Name a problem manager, which in a small team is a part of someone's role, to keep the list current, chair a short regular review and report on trends. Invite the engineers who responded to the incidents, because they hold the detail. Keep the review short and focused on decisions: what is the next action, who owns it and when will we look again?

Measures worth keeping

• The number of repeat incidents from the same cause, trending downward over time.
• The number of open known errors and their age.
• The proportion of problems with a named owner and a next action.
• The time from the second occurrence to a recorded problem.

Problem management in a small team

A team of a few people does not need a problem board and a manager. It needs a habit. After any incident review, ask one question: is this the second time? If it is, create a problem record and give it an owner. Once a month, spend twenty minutes reading the open problems and known errors aloud. For each, say the next action and who owns it, or decide to accept it and write down why.

Resist the temptation to wait for a perfect investigation. A problem record with a workaround and a plausible theory is already worth more than none, because it spares the next responder from starting from nothing. Refine the record as you learn. When a theory is disproved, say so in the record, so that nobody repeats the same line of inquiry.

Finally, protect the time. Problem work competes with feature work and loses, because the incidents it prevents are invisible. Put the actions into the same backlog as everything else, with the same visibility, and count the incidents avoided when you report results.

A problem record template

• Title: the symptom in words a responder would search for.
• Owner and status: who is accountable and whether it is open, under investigation, a known error, resolved or closed.
• Related incidents: links, with dates.
• Impact: how often it occurs, how many customers are affected and the cost to the team.
• Contributing factors: what is understood, and what is still unknown.
• Workaround: numbered steps and the time they take.
• Permanent fix options: each with effort, risk and benefit, and the one chosen with the reason.
• Actions: each with an owner and a due date, tracked as tasks.
• Next review date: the day someone will look again.

Problems and risk

A known error is a risk that has already shown itself. Treat open known errors as part of your risk register. A short list of the oldest and most severe, reviewed monthly with the service owners, keeps them from becoming permanent features of the landscape. Where the owner decides not to fix a problem, the decision is a risk acceptance, and it should be recorded with the person who accepted it and the date it will be reconsidered.

Common mistakes

• Skipping the problem record because the incident is closed.
• Naming a person as the cause.
• Leaving a known error with no owner, so that it never moves.
• Fixing the symptom each time and calling it a fix.
• Treating every incident as a problem. Open a problem when there is recurrence or real uncertainty about cause, not for every incident.