Cloud resilience tests should include the services needed to recover

During the AWS disruption in Northern Virginia on 7 December 2021, running EC2 instances continued operating while APIs used to launch new instances suffered errors and latency. Existing Route 53 DNS answers also continued, while customers had difficulty changing DNS records. AWS describes these differences in its published event summary.

For a cloud platform lead, the distinction is crucial. A service can keep serving current traffic yet lose the operations it would need to replace failed capacity, scale or redirect users. A test that checks only whether the homepage remains available will miss that weakness.

Resilience investment should follow the service’s limiting dependency. Another region can be useful, but only if the route into that region, the required data and the people operating recovery remain available under the failure being considered.

Trace the normal path and the recovery path

Choose one critical business journey, such as submitting an order or retrieving a customer record. Trace its identity provider, DNS, network path, application runtime, data stores and third-party services. Then draw a second dependency list for recovery: administrator access, deployment tooling, secrets, monitoring, change APIs and communications.

Compare the lists. A backup application in another region may still need the same identity service for new sessions. A failover procedure may require a DNS update or a control-plane operation that is unavailable in the chosen scenario. A dashboard may depend on the same network path it is meant to diagnose.

AWS’s account of the 2021 event also describes monitoring impairment that made diagnosis harder. The lesson is to test the evidence and access responders require, alongside the customer-facing components. This is a conclusion about dependency design, not a claim that a historical incident predicts a particular provider’s next outage.

Ask different questions of existing and new work

Consider a hypothetical two-region customer portal with a shared external identity provider. Its current sessions remain usable during a brief authentication interruption, but new customers cannot sign in. Moving application traffic to the other region would not, by itself, supply the missing authentication service.

A useful test separates several observations: whether existing sessions continue, whether a fresh login succeeds, what happens when a session needs renewal, and whether an administrator can reach the recovery controls. Use the actual configured session behaviour rather than assuming every identity implementation behaves the same way.

Define acceptable degradation with the service owner. Existing customers completing work while new logins fail may be tolerable briefly for one service and unacceptable for another. A read-only mode might protect information while preventing orders; that trade-off needs a business decision.

The same distinction applies to capacity. Test a running instance, a replacement instance and an attempted scale-out as separate cases. Recovery that depends on creating resources is different from recovery using capacity already provisioned and ready.

Design the experiment around a bounded question

Begin in an isolated or representative test environment where possible. State one hypothesis, the dependency being disrupted, the permitted resources, maximum duration and expected user effect. Identify who can authorise, stop and recover the exercise. A production test needs a reviewed scope, customer-impact limits and the relevant service owners present.

For the portal, the hypothesis might be that loss of fresh authentication produces a clear, bounded failure for new users while existing permitted sessions and administrative recovery remain usable. That is more testable than “the service is highly available”. Avoid adding unrelated faults until the team understands the first result.

AWS Fault Injection Service supports stop conditions based on CloudWatch alarms. Its documentation starts with a defined steady state and uses alarm thresholds to stop an experiment when behaviour becomes unacceptable. That mechanism is useful only when the chosen alarm observes the consequence that matters.

Alongside infrastructure metrics, watch a synthetic or controlled business transaction, error rate, data integrity and the availability of the stop route. If the telemetry needed to judge impact disappears, define in advance whether that loss itself ends the test. Assign a person an independent means of recognising trouble.

Observe the return to service

Keep the recovery procedure separate from stopping fault injection. Removing a fault may leave queued work, stale state, exhausted connections or a component needing operator intervention. Test the actions required to return the whole journey to its agreed state.

Record when the customer journey first fails, when a degraded mode becomes usable and when normal service is restored. Inspect pending work for loss, duplication and unexpected retries. For an order service, successful HTTP responses are insufficient if customers receive duplicate orders or the fulfilment system never sees them.

Use a DNS exercise to distinguish a failure to resolve a name from an inability to change records. Use a third-party dependency exercise to distinguish a clean refusal from a slow response that consumes application capacity. These scenarios can require different remedies even when customers see the same error page.

Finish by funding the specific limitation the test exposed: pre-provisioned capacity, an alternative access route, better degradation behaviour or a simpler recovery procedure. A second region should earn its place through the failure it addresses. Connect the result to recovery targets measured at the working service, then repeat the relevant exercise after the remedy is implemented.