Observability platform evaluation: replay the incident your team could not explain

A difficult incident is a useful procurement asset. The unresolved gap in a service review tells an operations lead what the next observability platform needs to improve: perhaps nobody could isolate the affected customers, link a slow request to a dependency or establish whether a deployment caused the change.

Use that gap to shape the evaluation. Give shortlisted suppliers an investigation with the same service behaviour, comparable signals and the access your responders would actually have. The result should tell you whether the proposed platform helps the team reach a defensible diagnosis, and what it costs to make that possible.

The exercise also separates two investments that are easily confused. Your organisation may need better instrumentation in the application, a better way to investigate the resulting evidence, or both. Use the distinction between monitoring and observability to identify which questions your team cannot currently answer before signing a platform contract.

Build the investigation around a decision

Here is a synthetic scenario to adapt to a permitted test environment. Checkout latency rises after a release, but only some customer requests are affected. The application calls a database and an external payment service. Responders must establish which stage contributes the delay and whether the evidence supports rolling back the release.

Start with the symptom rather than the answer. Let each candidate show how a responder would narrow the affected population, follow a slow request and compare the relevant behaviour before and after the change. Keep a known explanation with the exercise organiser so plausible storytelling is not mistaken for diagnosis.

The OpenTelemetry observability primer describes a distributed trace as the path of a request across services, made up of spans representing operations. It also explains why logs become more useful when correlated with trace context. In this exercise, that context is the bridge between a customer symptom and the component the team might change.

Three questions are enough to keep the demonstration anchored:

1. Which requests and users are affected, and which evidence establishes that boundary?
2. Where is the extra time spent along the request path?
3. What supports the proposed action, and what remains unknown?

Ask the supplier to distinguish an observed relationship from a causal conclusion. A deployment marker beside a latency spike is a lead to investigate. It does not, by itself, show that reversing the deployment will fix the problem.

Run it again with an important piece missing

After the first pass, remove a useful signal or break one correlation in the test environment. Perhaps a service does not propagate the trace context, or an external dependency supplies only timing information. Can the responder identify the missing evidence and still reach a narrower, useful conclusion?

Score that behaviour explicitly. An honest boundary around the diagnosis is more useful than a confident explanation assembled from unrelated events. The platform should also make the remediation cost visible: does closing the gap require an application release, a collector change or a different integration?

Then repeat a part of the investigation using the team’s normal access. Administrator visibility can make a demonstration look complete while the on-call engineer remains unable to query the relevant data. Conversely, broad access that reveals sensitive payloads can make the proposed workflow unsuitable. Include the permissions the production service would use.

Sampling belongs in the acceptance test

OpenTelemetry’s sampling guidance distinguishes early, head-based decisions from tail sampling that considers all or most spans in a trace. Head sampling alone cannot guarantee retention of every trace that later contains an error. Tail sampling can select using later evidence, but it adds processing and operational considerations.

That makes a supplier’s proposed sampling configuration part of the product being evaluated. Re-run the important diagnostic questions with the configuration used in the cost estimate. A demonstration with every trace available should not justify a purchase priced on keeping a much smaller sample.

There is a further consequence when traces generate metrics. Grafana Tempo’s documentation explains that metrics generated after ingestion reflect the traces that reached Tempo. If a collector has already discarded much of the data, derived request rates, errors and service relationships may describe that sample rather than all traffic. Grafana documents generating metrics in the collector before sampling as an alternative path.

For the operations lead, the question is therefore where each displayed measure comes from. Ask the supplier to trace one important metric back through collection and sampling. If a rare failure is the reason for buying the platform, demonstrate that the proposed telemetry policy preserves enough evidence to investigate it.

Buy the investigation you can afford to keep

Bring the commercial discussion into the same exercise. Estimate telemetry volumes from representative traffic, then show how the proposal changes during a burst. Ask which quantities drive charges, which data remains immediately searchable and what it takes to retrieve older evidence.

The retention period should follow the team’s investigation needs. If a customer reports a problem several days later, a successful live demonstration has not established that the necessary detail will still exist. Test a historical question at the age of data your team expects to use, including any restoration delay or additional charge.

Include the work outside the subscription: instrumentation changes, collector operation, access administration and migration of important queries. Where export is a requirement, demonstrate a useful export and show how another tool would interpret it. A file arriving successfully is not the same as preserving the investigative context.

End the evaluation with your own responder leading the exercise. Give comparable training time across candidates and keep supplier assistance visible. This does not need to become a speed contest. The stronger outcome is a sound diagnosis with clear evidence, understood gaps and a support burden the team can sustain.

Before inviting the shortlist, choose the service-level objective that makes the incident worth investigating. It gives every candidate the same starting point: explain the behaviour that matters to the service’s users, using the telemetry and budget the organisation can actually maintain.