Rollback Plan: How to Write One Before You Need It

A rollback plan describes how to return a system to its previous working state if a release causes harm. It is written before the release, reviewed by the person who would carry it out, and attached to the release record so that it can be found when it is needed.

The reason to write it in advance is practical. During an incident, people are under pressure, short of sleep and working from partial information. A plan written in those conditions is likely to be incomplete. A plan written beforehand can be read, questioned and tested.

What a rollback plan contains

• The trigger: the observable condition that causes a rollback, stated in numbers or behaviours where possible. For example, an error rate above a stated level for a stated time, or a failed check on a named critical journey.
• The decision maker: one named person who can call the rollback, and a named deputy.
• The steps: an ordered, specific list that someone unfamiliar with the release could follow, including commands, locations and expected output.
• The data position: what happens to data written since the release, and whether it is preserved, converted or lost.
• The verification: how you will confirm that the system has returned to a healthy state.
• The time limit: how long the rollback may take, and what happens if it exceeds it.
• The communication: who is told, by whom and through which channel.

Roll back or fix forward

Rolling back returns to the previous version. Fixing forward ships a new change that corrects the problem. Neither is always right.

Prefer rollback when the previous version is known to be good, the rollback is quick and low risk, and the cause of the fault is not yet understood. Prefer fixing forward when a rollback is not possible, for example because of an irreversible data change, or when the fix is small, understood and quicker than the rollback.

State your default in the plan. Teams that debate the choice for twenty minutes during an incident have lost twenty minutes. A common default is to roll back unless there is a specific reason not to.

Data changes are the hard part

Application code is easy to roll back because the previous build still exists. Data is harder. A migration that drops a column, rewrites values or changes a format may not be reversible, and new code may have written data that old code cannot read.

Reduce the risk by making data changes in compatible steps. Add before you remove. Deploy code that can handle both the old and the new shape, migrate the data, and only later remove support for the old shape. This is sometimes called an expand and contract approach, and it lets code roll back without a data rollback.

Where a destructive change is unavoidable, take and verify a backup first, state in the plan how the backup would be restored and how long that would take, and treat the release as a change that is not freely reversible. Mark it as such in the release record and require the sign-off of someone who owns the data.

Rehearse it

A plan that has never been run is a hypothesis. Rehearse the rollback in a staging environment before releasing anything risky, and time it. The measured duration is what you put in the plan, not an estimate.

Also rehearse the decision. In a short exercise, give the release owner a simulated trigger and see whether the right person calls the rollback within the time the plan allows. The most common weakness is not technical. It is hesitation, because nobody is sure who is allowed to decide.

Record it with the release

Keep the rollback plan with the release so that anyone can find it from the release itself. In Atlas, you can add a required operations check that confirms the plan exists and has been reviewed, so the release cannot move to Released while it is missing. A released release can be rolled back, a deployment can be recorded as rolled back, and a rolled back release can return to In progress, so that the fix is tracked against the same history.

Atlas does not perform the rollback. Your pipeline does that. Atlas holds the decision, the reason and the record.