Incident Communication: What to Tell Customers and When

During an incident, the people affected want three things: to know that you are aware, to know what they should do, and to know when they will hear more. Communication is not a courtesy added after the technical work. It is part of the response, with its own owner and its own schedule.

Know the audiences

• Responders: need the live detail, in the coordination channel.
• Internal stakeholders such as leadership, support and sales: need the impact, the current status and the next update time, in plain language and without jargon.
• Affected customers: need to know what is wrong from their side, what to do, and when you will update.
• Regulators and contractual counterparties, in some cases: may require notification within a stated period. Decide with your own counsel which obligations apply to you.

Choose an owner and a cadence

Assign a communications lead so that the commander and the engineers are not drafting messages. Set a cadence by severity, for example every thirty minutes for the most serious incidents and every few hours for lesser ones, and keep to it even when there is nothing new. "No change since the last update; we are still investigating the database" is a valid and useful update, because it tells the reader that you are working and when to look again.

What each message contains

• What is happening, in terms of the customer's experience rather than your architecture.
• Who is affected, and who is not, if you know.
• What you are doing about it, at the level of detail that helps the reader.
• What the reader should do, including any workaround.
• When the next update will come.

Templates

Initial message: "We are investigating reports that [what customers experience]. This affects [who]. We will post an update by [time]."

Ongoing update: "We have identified [the cause, if known, in plain terms] and are [action]. [Who is still affected.] [Workaround, if any.] Next update by [time]."

Mitigation: "Service has been restored for [who] as of [time]. We are monitoring and will confirm when the work is complete."

Resolution: "This incident is resolved. [Short summary of what happened and the effect.] We will share a summary of what we are changing by [date]."

Rules for the words

• State facts, not guesses. If the cause is not confirmed, say it is under investigation.
• Do not promise a time to fix unless you are confident. Promise a time for the next update instead.
• Do not blame a vendor, a team or an individual in a public message.
• Use the same terms the customer sees in the product.
• Apologise once, plainly, where you owe it, and then be useful.

Approval before sending

An inaccurate message to customers is hard to retract. Require that someone other than the author approves the text of customer updates, and keep the approval tied to the exact words.

In Atlas, customer updates go by email to the affected customers' primary contacts, taken from the CRM account, and nothing reaches a customer without someone approving that exact text. The subject and body that were approved are recorded together as a fingerprint that is checked again at the moment of sending. Editing an approved update revokes the approval and tells you it has. A customer with no usable contact, or whose address is suppressed, is named after sending as one who could not be reached and is not marked as told. If nobody can be reached the update is not sent. Under every sent update, Atlas lists each customer by name with the state of their email, from waiting to send to delivered, bounced or marked as spam.

Atlas does not host a public status page for your own services. If you need a page that customers check, you must publish it separately. The Atlas status page at atlas.wrxstack.com/status reports on the Atlas platform.

After the incident

Follow the resolution message with a short written summary once the review is done: what happened, what the effect was and what is being changed. Customers who experienced a serious failure value a specific account of the remedy more than an apology.