DNS resilience: test resolution before the cache runs out

The website still opens on an engineer’s laptop, so DNS is marked healthy. A new customer cannot reach it. The laptop has an answer cached; the customer needs a fresh lookup. The same hostname can therefore appear available and unavailable during the same incident.

DNS resilience needs tests that distinguish an answer already held by a client or resolver from one that can be obtained now. It also needs an operating plan for changing records when the usual administration route is disrupted.

Draw three separate dependency chains

Start with authoritative DNS: the servers responsible for answering for the organisation’s zones and the delegation that leads resolvers to them. Then map recursive resolution: the resolvers used by staff and workloads, including forwarders for private services. Finally, map administration: registrar access, DNS management, identity, approvals and deployment automation.

These chains can fail independently. Public authoritative DNS may work while an office forwarder is unavailable. An authoritative service may continue answering while administrators cannot sign in to change a record. Calling all three “DNS uptime” hides the distinction.

The long-standing guidance in RFC 2182 stresses geographic and network diversity for authoritative servers. Its central lesson remains useful: several servers do not provide meaningful independence if one site, power supply or network failure can isolate them all.

Ask what redundancy actually protects

Review the failure domains behind the design, including provider infrastructure and the process that distributes changes. A second DNS provider can reduce some dependencies but adds work: records, routing features, signing arrangements and operational procedures must remain consistent.

Do not assume every proprietary record or traffic-steering feature transfers unchanged. Before adding a provider, test the actual zone behaviour and establish how updates reach each service. A second provider serving an old or incorrect answer can prolong an incident.

For internal resolution, check whether nominally separate forwarders share a host platform, network route or upstream resolver. Include cloud and remote-user paths. The private-endpoint DNS design should identify which client populations rely on each forwarding route.

Test the cache boundary deliberately

A time-to-live is part of the caching design, not a promise that every user will switch at an identical instant. Applications and resolvers have their own behaviour, and existing connections can continue without another lookup. Test the clients that matter to the service.

RFC 8767 describes serving stale DNS data to improve resilience when fresh data cannot be obtained. Where a resolver supports and enables that behaviour, a successful answer may be older than expected. Record the resolver configuration and answer state rather than treating success as proof that authoritative DNS is reachable.

Prepare a controlled test name in a non-production zone or an agreed test area. Compare a cached query with a previously uncached query through representative resolvers. Change a test record, observe when clients receive the new answer and record what happens if the test authority or forwarding path is unavailable. Keep the test isolated from live customer names.

Use an exercise that covers more than the public website

Include private-endpoint resolution from cloud and office clients. Public authoritative DNS can remain healthy while an internal forwarding rule or private zone prevents the application from connecting.

Define success in terms of a usable service. DNS recovery is only one stage; clients still need a reachable destination, valid security configuration and a working application. If a failover answer points to an unready environment, resolution has recovered but the service has not.

Keep the route to repair independent

Document where the zone configuration, account ownership and recovery instructions are held. Make sure the team can retrieve them if the normal corporate application is unavailable. Protect emergency access, test it periodically and keep changes attributable to a named person.

Set alerts for the conditions that matter: inconsistent answers, resolution failure from relevant locations, expiring operational dependencies and failed configuration deployments. Assign an owner for each alert and an action the owner can take.

After the exercise, record the last successful fresh answer, the first observed failure and the first successful transaction after recovery. Compare that service timeline with the target in the recovery plan. The result should explain what remains available, for whom and for how long, without relying on a fortunate cached answer.