AI agent evaluation: make the purchase request fail halfway

The most revealing moment in an AI-agent demonstration may come after the purchase request has been created, when the connection drops before the agent receives confirmation. Does the system find the existing request, submit another one, or leave an operator guessing?

For an enterprise AI lead choosing a purchasing assistant, that question connects model performance to a real operating cost. A duplicate request can create approval work, supplier confusion and reconciliation effort. A stalled request can delay a project. The agent’s ability to explain itself matters, but the buying decision needs evidence about the business transaction.

Use one bounded workflow as the unit of evaluation. Define the permitted action, the systems it touches and the state in which a human can safely take over. Then compare suppliers on the same normal and interrupted runs.

Separate the plan from the authority to execute it

A sensible purchasing plan does not establish permission to create an order. Specify which legal entity, cost centre, supplier and spending range the workflow may use, and which changes need approval. Ask where those restrictions are enforced when a tool is called. An instruction in the prompt is not a substitute for a target-system permission or an application rule that rejects an out-of-scope request.

Amazon Bedrock offers a useful architectural example. Its return-of-control option sends the proposed action inputs and an invocation identifier back to the developer’s application. That application can invoke the action and return the result to the agent. This creates a place to validate the request before execution; it does not remove the developer’s responsibility to implement that validation.

For the evaluation, ask the supplier to show the rejected transaction as well as the accepted one. Can a request for an unapproved supplier reach the purchasing system? What happens if authority is withdrawn while the agent is waiting? The useful evidence is the enforcement result and the affected business record, not a reassuring refusal in the conversation.

The timeout that distinguishes the candidates

Consider this hypothetical test: the agent creates a £4,000 purchase request, the purchasing system commits it, and the acknowledgement is lost. The user then asks the agent to try again. There is already one valid request, but the agent does not yet know that.

A robust design needs a way to recognise the same business operation across attempts. Ask whether an operation identifier survives retries, restarts and a handover to a different worker. Check how the application looks up an uncertain result before creating something new. A fresh conversation identifier for every attempt will not, by itself, identify the original purchase.

Stripe’s API documentation illustrates why the downstream contract matters. For API v1, a retry with the same idempotency key returns the saved first response, including a 500 error. Keys can be removed after they are at least 24 hours old; reusing one after removal creates a new request. The lesson for agent buyers is to establish the actual retry semantics and retention window of each connected system. “Supports idempotency” needs an implementation behind it.

If the purchasing application has no suitable duplicate-protection facility, the integration may need its own durable operation record and reconciliation process. Include that work in the proposal. Do not let it disappear into a vague promise that the model will recognise what it has already done.

Resuming the agent and recovering the business are different jobs

LangGraph’s checkpointers save graph state and support recovery from interrupted execution. Its pending-write mechanism can avoid rerunning nodes that completed successfully when another node failed in the same step. Those are useful capabilities to inspect in a supplier’s architecture.

For the purchasing decision, examine the boundary beyond that stored state. Restoring an earlier agent checkpoint does not establish that an external purchase request has been cancelled. The purchasing system remains the authority on whether the request exists, has been approved or has already triggered further work.

Give each candidate the same short set of proposed failure tests in an isolated environment:

• Acknowledgement lost: the transaction succeeds but its response is withheld. The expected outcome is one business request and an explainable status.
• Partial completion: the request exists but attachment upload fails. Recovery must resume or repair the missing step without creating another request.
• Permission withdrawn: execution resumes after access changes. The current authority must determine whether the next action proceeds.
• Cancellation requested: an operator stops the workflow after a downstream change. The system must distinguish stopping future actions from undoing completed ones.

Agree the expected target-system state before the demonstration. Otherwise, a supplier can label almost any graceful-looking error message a successful recovery.

Buy the workflow you can operate

Ask the operator who would inherit a failed run to reconstruct it from the evidence. They should be able to connect the user request, proposed action, approval, execution attempt and downstream record without searching several unrelated consoles by guesswork. Assess whether the recovery procedure requires an ordinary service operator, an integration engineer or the supplier’s support team. Those are materially different running costs.

Keep a conventional workflow in the comparison where inputs and routing are predictable. An agent may earn its place by handling variable requests or resolving incomplete information. It need not own every subsequent transaction. A useful design can let the agent prepare a request while deterministic application logic controls execution and reconciliation.

Select the candidate that completes the valuable task within the agreed authority and leaves recoverable failures at an affordable operating cost. Where the workflow depends on a human decision, the next design question is how approval controls the action that actually executes.