Incident review actions should change the next failure

The hardest part of an incident review is choosing which work should displace next week’s delivery commitments. Updating a runbook takes little time. Changing a shared dependency may take a quarter. Both can be reasonable responses, but an IT operations lead needs a better basis for choosing than the length of the action list or the seniority of the person proposing it.

Start with the effect you want to change: fewer occurrences, fewer customers affected, earlier detection or faster recovery. An action earns its place when the team can explain how it produces that effect and how it will check the result.

A recurrence does not mean the previous work failed

Google’s published postmortem case study describes a rack-decommissioning operation whose automation bug caused satellite machines around the world to be erased. An empty filter was interpreted as no restriction on the machines selected. The account says a similar incident happened three years later, but the changes made after the first event substantially reduced its spread and rate.

That distinction matters when funding reliability work. Some changes prevent a particular trigger; others limit damage when a trigger escapes. Judging every action solely by whether an incident ever recurs would undervalue containment and recovery improvements.

Google’s guidance also calls for action owners, priorities and verifiable end states, with attention to weaknesses in the system rather than blame directed at individuals. Those are useful foundations. The next management decision is which of the proposed changes deserves scarce engineering time.

Turn the finding into a causal argument

For each finding, write a short explanation connecting evidence, mechanism and proposed change. Keep uncertainty visible. “The dependency became slow before requests accumulated” is an observation; “retry traffic caused the overload” is a hypothesis until the traces, configuration or a controlled reproduction support it.

A good action might say: restrict retries on this call path so that a slow dependency does not consume all available request capacity; validate under a representative failure condition before enabling the change widely. That is specific enough to challenge. If retries were already bounded, the proposed work may miss the cause.

Documentation can be the right answer when responders lacked a necessary fact. It is a weaker answer when the same destructive operation remains easy to perform and the only protection is remembering a warning. Compare the cost and expected effect of changing the operation, limiting its scope, improving detection and rehearsing recovery. Do not assume the most architectural proposal is automatically the best investment.

Three possible responses to one outage

In this hypothetical example, a retailer’s checkout slows when its stock service responds intermittently. The review establishes that waiting requests consume checkout capacity. Engineers propose increasing timeouts, adding an alert and isolating the stock-service call path.

A longer timeout could reduce premature failures if the dependency normally completes slightly late. It could also leave more requests waiting. Approving that change requires a test showing what happens to capacity and customer completion under the observed failure condition.

An alert could shorten the time before intervention, provided a responder has an effective action. If the only response is to watch the same dashboard, another page does little to reduce impact.

Isolation could preserve capacity for other checkout work, but its fallback must make commercial sense. Continuing without a stock confirmation might create orders the retailer cannot fulfil. The service owner needs to decide which customer outcome is acceptable before engineers implement it.

These proposals address different mechanisms. They can be combined, deferred or rejected explicitly. The review should not quietly turn all three into equally urgent tickets.

Fund a small, ordered backlog

Put actions that address an ongoing dangerous condition at the front. Then compare repeat exposure, customer consequence, confidence in the diagnosis, engineering effort and dependencies. Where uncertainty prevents a sensible choice, fund a bounded investigation with a decision date. Avoid multiplying those inputs into a precise-looking score unless the organisation can defend the scale and weights.

For each selected action, capture the finding, expected effect, priority rationale, one accountable owner and evidence required for acceptance. A supplier or another team can own a dependency, but someone in the service organisation still needs to own progress and escalation. Record the consequence of deferral so that unstaffed work remains a conscious risk decision.

Distinguish “implemented” from “effect checked”. A deployed change is implementation evidence. A controlled failure exercise showing that unrelated requests continue to complete is evidence about isolation. Production observation can add confidence, but a quiet week alone does not demonstrate that a rare failure has been prevented.

Use the next reliability planning session to examine the oldest high-consequence actions and repeated findings across incidents. Reopen work whose acceptance condition was never tested; retire proposals whose underlying assumption has been disproved. If the chosen action is better detection, connect it to an alert with a clear owner and response. The result should be a shorter backlog the team can defend, fund and finish.