Service credential rotation: include the integration that only runs at month-end

The nightly integration passes its test. The application stays healthy. The old credential is retired. Three weeks later, the month-end export fails because it was still using a copy nobody found.

In this hypothetical scenario, the identity lead did rotate the credential successfully for the systems that were tested. The failure was in the definition of the change: an infrequent consumer was outside it. For a critical integration, rotation needs to cover the software’s working calendar as well as the secret’s expiry date.

Find the consumers that a normal day misses

Start with the credential’s purpose and the service that accepts it. Then trace how each application, job and external integration obtains the value. A reference to a central secrets service is different from a value copied into a scheduler, deployment variable or supplier-managed configuration. Updating the central copy will help only the consumers that actually retrieve it.

Ask when retrieval happens. A process may load a credential at startup, fetch it for each connection or retain it in a cache. Establish the behaviour from the application’s documentation and a controlled test. The deployment plan needs to say which processes must refresh or restart, and what happens to queued work while they do.

Include month-end reporting, disaster-recovery environments, rarely used administrative scripts and integrations maintained outside the application team. Do not assume a lack of recent use means a credential has no remaining business purpose. A quiet consumer can still be critical when its scheduled work arrives.

For each one, identify a representative transaction and a way to exercise it without creating unwanted business effects. The month-end export may need a controlled test with an agreed destination, rather than waiting until the actual financial close to discover whether authentication works.

Choose the sequence the target system supports

Overlapping credentials can make a staged change possible. A system that accepts only one password at a time needs a more coordinated cutover. These are different change plans, and a secrets-management product does not make their underlying constraints disappear.

AWS documents an overlap procedure for IAM user access keys: create a second active key, update applications, inspect use of the old key, then deactivate it before eventual deletion. The deactivation stage allows the team to test with only the new key working and, where appropriate, reactivate the old one if a missed application emerges.

That example shows why “rotate” should name the intermediate states. The team needs to know when both credentials work, when only the replacement should work and when recovery through the old key is no longer available. It is not a universal recipe for database passwords or every API provider.

A useful change plan gives the overlap window a purpose and an end. Keep it long enough to exercise the required consumers, but do not turn uncertainty about ownership into indefinite acceptance of both credentials. If a delayed job cannot be tested, resolve that explicitly with its owner before retiring the credential it may depend on.

A successful connection can prove the wrong thing

Database rotation exposes another trap. In AWS Secrets Manager’s Lambda-based database rotation strategies, single-user rotation does not drop existing database connections. AWS also documents a short interval between changing the database password and updating the secret, during which authentication can be denied.

A healthy application using an already-open connection therefore does not establish that it can authenticate with the replacement. Test a fresh connection through the real credential retrieval path. Avoid making the application look healthy by testing with a manually supplied secret that bypasses its normal configuration.

Secrets Manager’s alternating-users strategy has different semantics: it rotates between an original and cloned database user, and both sets of credentials remain valid after rotation. AWS notes that subsequent permission changes to the original must also be made to the clone. This is an availability design with two identities to maintain, not immediate invalidation of every previous credential.

Choose between these patterns using the application’s connection behaviour and the database’s permission model. For the identity lead, the question is whether the proposed procedure delivers the intended security outcome without concealing an access path that stays live.

Separate recovery from retirement

Prepare the recovery path before the cutover. If authentication fails, can the deployment revert, can the previous credential be re-enabled, or must the team issue another one? Verify the permitted mechanism in the specific service. A rollback that assumes a deleted secret can be retrieved is not a working recovery plan.

Routine rotation and suspected credential compromise also need different decisions. Re-enabling an old credential may be an acceptable recovery step during a planned change; it may be inappropriate when that credential is suspected of being exposed. In that case, coordinate containment and service restoration instead of quietly restoring the very access being removed.

Close the change with two distinct pieces of evidence: intended consumers can perform their work using the replacement, and the retired credential fails at the intended authentication boundary. Use a controlled negative test where safe. Also establish whether existing sessions or tokens require separate action; invalidating a credential and ending previously established access need not be the same operation.

Keep secret values out of tickets and logs. Record the credential identifier, affected consumers, test results and retirement state through approved systems. If finding the consumers was the hardest part of the change, the next improvement belongs in the ownership and lifecycle of non-human identities.