Monitor an AI service for the failures an uptime chart cannot show

The assistant is answering quickly. Its error rate is low. Yet staff have started copying obsolete instructions into customer replies. For the enterprise AI lead, this is a production failure even though the infrastructure dashboard is green.

Monitoring an AI service means watching the work it performs as well as the systems that deliver it. A knowledge assistant depends on source documents, retrieval, permissions, prompts and a model. Those components can change independently. The operating plan needs to reveal which dependency has failed and give someone the authority to restrict the service before more users rely on it.

Use a failure with an observable consequence

Consider a hypothetical internal support assistant. A policy team updates the authoritative refund procedure, but a connector stops refreshing the search index. The assistant continues to cite the old procedure. More compute capacity would not help, and switching to a larger model could produce the same obsolete answer at greater cost.

For this service, “working” must include retrieving the currently approved procedure. Track the source publication time, successful ingestion time and version available to retrieval. Add a small set of questions whose expected evidence changes when important policies change. This separates a stale knowledge problem from a generation problem: did retrieval return the wrong document, or did the model misrepresent the right one?

Freshness needs a business boundary. A facilities directory and a procedure governing same-day customer refunds may tolerate different delays. The source owner should define when an update becomes operationally binding; the AI lead should ensure the assistant can meet that requirement or withhold the affected answers.

Give each signal a different job

Availability and latency belong to the service engineering team: can users submit work and receive a response within the time the workflow allows? Measure the whole interaction, including retrieval and tool calls. Streaming an initial token quickly does not establish that the completed answer arrived in time to be useful. Use service-level objectives tied to the user journey for this part of the service.

Task quality belongs to the product owner with domain reviewers. For the support assistant, useful measures might include whether the answer is supported by the cited procedure, whether it omits a required step and whether a reviewer must substantially rewrite it. Separate simple lookups from complex exceptions. A changing mix of questions can otherwise look like an improvement or deterioration in the model.

Security and harmful behaviour need their own escalation path. A suspected disclosure to an unauthorised user should not wait for a weekly average to cross a threshold. Preserve the necessary evidence, restrict the affected capability and involve the incident owner. Budget overruns, by contrast, may justify rate limits or a narrower feature while an engineering team investigates. One aggregate “AI health” score obscures these different decisions.

Verify what the monitoring system can see

Do not assume the model provider has captured the evidence required to investigate. In Amazon Bedrock, invocation logging is disabled by default. AWS documents support for specified calls through the bedrock-runtime endpoint; requests through bedrock-mantle are outside that logging mechanism. A service using more than one route therefore needs an explicit coverage check.

Link each application request to the prompt version, model identifier, retrieval result identifiers, permission context and downstream outcome where available. Avoid making full sensitive prompts the default answer to every observability problem. Decide which content investigators need, which metadata is sufficient, who can access it and when it should be removed. Test that an investigation remains possible under those limits.

Also monitor missing evidence. An empty failure chart may mean nobody is failing; it may mean the collector has stopped. Count expected requests, recorded traces and completed evaluations separately. Alert on a widening gap between them.

Sample deliberately, then check the sample

Automated evaluation can help review more production interactions, but it is another system with dependencies. LangSmith’s online evaluator documentation describes filters and sampling rates. It also says that reaching an evaluator’s weekly spend limit pauses evaluation until the limit resets or is increased. A quality dashboard needs to show that pause; yesterday’s score is not evidence about today’s unevaluated traffic.

Combine a random sample with targeted review of consequential questions, new workflows, complaints and failed tool calls. Sampling only negative feedback misses users who accept an incorrect answer without recognising the error. Sampling only completed interactions misses abandoned requests. Label the population each score describes.

Use domain experts to calibrate an automated judge against examples of acceptable and unacceptable work. Review disagreements rather than treating the judge’s confidence as proof. Preserve the judge configuration and version alongside results, because changing the evaluator can move the score without changing the service.

Rehearse the response to a bad answer

For each important signal, specify the owner, the action and the evidence required to resume normal operation. In the stale-policy example, the data owner repairs ingestion, the AI team temporarily suppresses the affected topic, and the product owner verifies answers against the new procedure before reopening it. The user-facing fallback should explain where staff can find the authoritative information.

Re-evaluate after changes to the model, prompt, source connector or access rules. Maintain comparable cases across versions, but also add production failures that the original pilot never exposed. This creates a feedback loop between live operation and release testing without assuming that every new model behaves like its predecessor.

Start with the failure that would cause the greatest avoidable business harm. Define its detection and recovery action, then build the evaluation cases that would expose it. That is the foundation of a monitoring plan an on-call team can actually use.