Write an incident response plan people can use during an outage

A dozen specialists can join an incident without making it easier to manage. In a case described in Google’s Site Reliability Workbook, GKE cluster creation was failing while existing clusters continued to serve traffic. After two and a half hours, the incident channel had a dozen participants and communication was becoming difficult. A handover brought an explicit command structure, with separate operations and communications leads. The responders also used a shared working document as technical discussion outgrew the chat channel.

For a CISO writing an incident response plan, the useful lesson is about coordination. Technical expertise needs a way to become an authorised decision, with a record the next shift can understand. That becomes harder when the organisation’s identity provider, chat or document system is itself involved in the disruption.

The plan should let people establish command, limit damage, obtain help and return services to operation. Detailed forensic and recovery runbooks support those decisions. They cannot replace agreement about who may interrupt trading, commit emergency expenditure or accept a constrained return to service.

Put authority beside the decision

Start with one business service and its owner. Identify the dependencies that matter to response, including identity, remote administration, communications and suppliers. Name the incident lead and deputy, and specify who temporarily holds command before either is reached. An escalation list without interim authority can leave the first responder waiting while the damage continues.

NIST SP 800-61 Revision 3, published in April 2025, replaced Revision 2 and integrates incident response into wider cybersecurity risk management. For the CISO, that framing connects preparation to business decisions made before the emergency: tolerable disruption, delegated authority and the resources available to respond.

Use a compact decision table in the operational plan. The structure below is a proposed starting point; the organisation must agree the roles and limits for its own service. Put the named primary contact, deputy and alternate contact route beside each accountable role in the controlled version.

Trigger	Decision and accountable role	Authority boundary and evidence
Credible compromise or serious service disruption	Incident lead declares the incident and establishes command.	Record affected services, known facts, uncertainty and declaration time. State who acts until the lead or deputy joins.
Continued exposure or suspected spread	Technical lead proposes containment; designated authority approves disruptive action.	Identify permitted systems and disruption limits, expected consequences and the conditions for reviewing or reversing the action.
Internal capability is insufficient	Incident lead requests external help through the authorised commercial contact.	Record engagement scope, spending authority, access and data-sharing limits, and the verified supplier contact.
Service is proposed for return	Recovery lead presents validation; business owner authorises the proposed return.	Record test evidence, residual risk, monitoring responsibility and conditions that would trigger another interruption.

Containment and recovery need separate decisions even when the same person approves both. Stopping suspicious access may be justified before the cause is fully understood. Restarting a service requires evidence about the proposed operating state and the dependencies it will reconnect to. A successful technical restart alone does not establish that customer transactions can proceed correctly.

For disruptive actions, agree the permitted scope in advance. Where the service touches industrial equipment, coordinate the IT incident handover with the people responsible for the physical process. Isolating a workstation, disabling a shared identity integration and shutting down an order service have different consequences. Where advance authority is limited, the plan must show the escalation route and what the responder can do while waiting. Avoid an instruction such as “disconnect everything” that transfers an undefined business decision to the person nearest the keyboard.

Make the plan survive its own failure scenario

A document stored only behind the affected identity provider is part of the outage. Maintain a controlled printed pack or protected offline copy on designated equipment, with a route to reach the required people that does not share the same dependency. Specify who holds each copy, how changes reach them and how obsolete versions are withdrawn.

Keep the offline pack small enough to use under pressure: command and contact details, decision authority, essential service dependencies, the route to technical runbooks and a way to start the incident record. It should identify the current version and its owner. Avoid copying privileged credentials into a broadly distributed response plan; emergency access needs its own controlled retrieval and review process.

The record should capture decisions, timestamps, owners, supporting observations and unresolved questions. Separate a working theory from an established fact. During a handover, the incoming lead needs to know what was changed, why, what remains exposed and when the next decision is due. A transcript of every message can bury those essentials.

Communications also needs an owner. Agree who consolidates status for executives, customers and suppliers so technical responders are not repeatedly interrupted for incompatible updates. Keep reporting obligations in a maintained annex owned by the appropriate legal or compliance lead, with routes for determining the obligations applicable to the incident.

Exercise the missing tools, then repair the plan

Run this hypothetical exercise: the corporate identity provider and incident chat are unavailable while an order-processing service is disrupted. The first responder has a phone and the controlled offline pack. Ask them to establish command, reach a deputy, obtain a containment decision and record the action. Then introduce an external-support requirement and hand command to a second participant using only the available record.

Watch where the process stops. A correct phone number does not prove that its owner has the authority to act. An approved emergency budget does not prove that someone can engage the supplier without the failed procurement login. A downloadable runbook does not help if its decryption key depends on the unavailable identity service.

Give each failure an owner and a retest. Update the plan when a critical supplier changes, a decision maker leaves, an identity dependency moves or an exercise exposes a blocked step. Review its usability with someone outside the authoring team: they should be able to execute the agreed scenario without informal guidance from its designers.

Where the exercise exposes a dependency on outside specialists, use the incident-retainer scope and authority checks to settle how that help will be engaged. The response plan becomes credible when those arrangements work during the failure it is meant to manage.