RTO and RPO testing: stop the clock when the service works

A database restore finishes at noon. The order-processing service remains unusable until the following morning because permissions and an integration have not been recovered. Calling that a two-hour recovery would give the business the wrong answer.

This hypothetical failure is a useful starting point for an infrastructure lead defending a recovery plan. The business needs to know how long work could be interrupted and which data might be unavailable. A component’s successful restore is evidence, but it is not the whole acceptance test.

Agree what the target means before the exercise

A recovery time objective, or RTO, sets the acceptable time to restore service. A recovery point objective, or RPO, concerns the acceptable age of the data recovery point: how far back the business can tolerate going. They answer different questions. Recovering quickly from an excessively old backup can meet a time target while failing the data-loss tolerance.

AWS Well-Architected guidance defines RTO from service interruption to restoration and recommends setting recovery objectives against business impact and dependent workloads. That boundary matters. Starting the business stopwatch only when an operator launches a restore hides detection, decision and access delays.

Choose an observable interruption event and an observable return to the agreed service level. Record incident detection, declaration, authorisation and the start of technical recovery as intermediate milestones. If a contract or existing internal measure starts at declaration, preserve that definition and report the full interruption alongside it. Do not present the narrower measure as the complete period of business unavailability.

Define the stop condition with the service owner. “Database available” may be necessary, but an order-processing service might also require application authentication, access to the correct records, successful creation of an order and confirmation that the downstream system receives it. Decide whether a degraded mode counts, its capacity limit and which work still cannot proceed.

Show the missing hours

In the same hypothetical example, assume a four-hour RTO and a 15-minute RPO. The service becomes unavailable at 08:00 on day one. Monitoring detects the failure at 08:15; recovery is declared at 08:45. Authorisation and recovery access are ready at 10:00, when the database restore starts. It finishes at 12:00.

The restore job took two hours. Yet the four-hour business recovery allowance has already been consumed, and the application still cannot operate. Permissions and a downstream integration are repaired by 08:30 on day two. Reconciliation and an accepted end-to-end transaction finish at 09:00. The observed service recovery time is 25 hours, exceeding the objective by 21 hours.

Those timestamps identify different interventions. The delay before restoration raises questions about authority and recovery access. The long delay afterwards points to missing application dependencies and restoration order. Buying faster storage would address only the component duration; it would leave the largest demonstrated gap intact.

For the data test, suppose the recovered database contains committed records through 07:50 on day one, with interruption at 08:00. The ten-minute gap is within the stated 15-minute tolerance for that database. It does not prove that the complete service has consistent data. A downstream system could contain transactions absent from the recovered database, or the reverse. Reconcile the relevant records and queues before claiming that the business data requirement passed.

Make validation part of recovery

A useful validation checks the recovered state and a representative business action. Confirm access through the intended identity, consistency across dependent systems and handling of work left in flight. Avoid testing only an administrator’s ability to connect; privileged access can conceal a problem ordinary users will face.

AWS Backup documents an event-driven validation workflow triggered after a restore testing job reaches COMPLETED. EventBridge can invoke a target such as Lambda, and the validation result can be reported back through the service API. The documented separation between restore completion and validation is useful: it gives you a place to test what the restored resource actually does.

The organisation still has to implement the right validation. A successful health check does not establish that reconciliation succeeded or that a customer-facing transaction can complete. Keep the individual component result and the broader service acceptance result visible, especially where the automated restore test covers only one resource.

Turn a failed target into a funded action

For each missed criterion, identify the dependency, the responsible owner, the proposed change and the evidence a retest must produce. In the example, that could mean rehearsing recovery access and restoring application permissions as part of the runbook, then testing the integration before accepting the service. Set a date for the retest and retain the original result.

Where the required recovery capability is unaffordable or technically unsupported, take the gap back to the business owner with options and consequences. Changing a target is a business decision; silently moving the clock or narrowing the tested service is not a substitute.

Use these criteria in a recovery exercise that tests the service and its dependencies. Repeat comparable scenarios after remediation, stating any changed scope. The evidence you need is a credible route from interruption to usable service, with acceptable recovered data.