Known Error Database: What to Record and How to Use It
A known error database holds the failures you already understand and how to get around them. Its value is measured at three in the morning, by whoever is on call.
A known error is a problem whose cause is understood and for which a workaround exists, although the permanent fix has not been made. A known error database, sometimes abbreviated KEDB, is the collection of those records. It saves time because a responder who recognises a symptom can find the workaround immediately instead of rediscovering it.
What each entry needs
- A title that describes the symptom as a responder sees it, not as an engineer understands the cause. People search by what they observe.
- Symptoms: the alerts, messages and behaviours that identify this error, with examples.
- Cause: what is understood about why it happens, and what is not.
- Workaround: numbered steps that someone unfamiliar with the system can follow, with the expected result of each.
- Impact and severity: the usual effect on customers, and how long the workaround takes to restore service.
- Fix status: the permanent fix, its owner, the date and a link to the work.
- Related incidents: links to the incidents where it occurred.
- Last verified: the date someone last confirmed that the workaround still works.
Make it findable
A record nobody finds is worthless. Link entries from the alerts that detect them, so the page carries the workaround. Add known errors to the incident process: the first responder asks whether the symptom matches a known cause. Keep the list searchable by symptom words, and keep titles in plain language.
In Atlas, recurring failures are recorded as known causes on the Incident reviews screen with a workaround. Link them in the alert text and in the runbook for the affected service.
Keep it current
- Verify each workaround on a schedule, for example every quarter, and update the verification date.
- Retire an entry when the permanent fix ships and has been observed to work.
- Review the open entries monthly. An entry that is old and unowned is a decision waiting to be made, either to fix it or to accept it formally.
- After each incident, ask whether the response used or should have used a known error, and update accordingly.
Known errors and runbooks
A runbook describes how to perform a procedure. A known error describes a specific failure and its workaround. The workaround often points to a runbook step. Keep them linked and avoid duplicating text, because two copies drift apart. See the guide on runbooks and playbooks.