Enterprise AI platforms: make the shortlist prove the workflow

The shortlist meeting is where an enterprise AI programme can quietly become three different projects. One supplier demonstrates an assistant inside office applications. Another offers models and retrieval services for developers. A third proposes an agent that can update business systems. All may be credible products. Decide which AI capabilities to buy, build or extend before treating these suppliers as interchangeable purchases.

For the enterprise AI lead, the immediate decision is which operating model can deliver a particular piece of work. Choosing a platform also chooses an integration burden, a route for controlling access and a set of people who will deal with failures. Those commitments deserve as much attention as the quality of the generated answer.

Decide what you are buying

Start with a workflow narrow enough to describe without naming a vendor. “Help account managers prepare for renewals using approved account documents” is a workable scope. “Give everyone AI” leaves almost every important requirement unresolved.

The renewal workflow has a user, a source of information and a destination for the output. It also has boundaries: an account manager should not discover another team’s restricted negotiations, and an assistant preparing a briefing should not silently change contract terms in the CRM. Whether the proposed product only drafts text or also takes action changes the evaluation.

Then separate the work the supplier delivers from the work your organisation must build. A model service may leave retrieval, permissions and the user interface to your developers. An embedded assistant may fit an existing workspace but leave less freedom over the interaction. An agent adds a further question: who controls its actions when connected systems disagree?

This is why a broad feature score can mislead. A long list of models does not compensate for an unsuitable deployment model. Equally, a convenient interface is worth little if the process still requires employees to copy information between several systems.

Permissions are a separate acceptance gate

Microsoft’s Copilot privacy and security documentation says the service surfaces organisational information that the individual user has permission to view. That documented boundary has a practical consequence for buyers: an assistant respecting existing permissions can still expose material that the organisation has shared too broadly.

Include two different checks in the pilot. First, can a user retrieve information they are not entitled to access? Second, do the permissions themselves reflect the business boundary you intended? A successful technical permission test does not resolve an overly permissive source library.

Use distinct test identities, including a user outside the relevant team. Change an entitlement during the trial and establish when that change takes effect in retrieval and generated answers. Ask the supplier to explain what happens to previously generated content. The answer should cover the complete workflow, including connected agents and exported documents.

Set the required result before testing. An unauthorised disclosure is a failed control, not a small deduction from an otherwise excellent average score. Where deployment region, retention or contractual data use is mandatory, assess the exact service configuration against those requirements as well. Our guide to enterprise AI data controls separates the questions that can otherwise get collapsed into a single security promise.

Give the trial enough difficult work to be useful

A pilot made entirely of tidy documents establishes very little about the workload waiting after launch. Include routine cases, conflicting versions, incomplete evidence and questions for which the permitted material contains no answer. The objective is to discover which failures the operating team would inherit.

For the renewal example, an acceptable output might identify the current contract end date, distinguish it from an old proposal, cite the supporting document and leave an unsupported discount assumption unresolved. A fluent briefing that blends the two versions should fail even if it looks ready to send.

Evaluation tooling can help organise that comparison. Amazon Bedrock documents automated, human and model-based evaluation approaches, including use of custom prompt datasets. Its RAG evaluation also separates retrieval and response assessment. That distinction matters: a poor answer may originate in the documents retrieved before the model writes anything.

These tools can produce useful measures, but the buying team still needs to define a satisfactory business result. A model’s score should not become the acceptance rule merely because it is easy to generate. Have people who do the work assess a sample of outputs against the same criteria, and investigate disagreements rather than smoothing them into one number.

Keep part of the evaluation material outside the supplier’s preparation pack. Give each candidate comparable configuration time, then test the held-back cases. Otherwise the exercise risks comparing how carefully three demonstrations were rehearsed.

Price the work that survives review

Review effort can reverse a seemingly obvious choice. Consider this hypothetical comparison for a task that always requires human approval. Platform A costs £120 to process 100 cases; reviewers spend six minutes per case, valued at £30 an hour. The combined processing and review cost is £420. Platform B costs £200 but needs three minutes of review per case, bringing that total to £350.

If both deliver 90 acceptable cases, the cost per accepted case is about £4.67 for A and £3.89 for B. The calculation includes the cost of failed cases in the numerator. It excludes integration, licences and ongoing support, which would need adding before an investment decision. The point is that a cheaper generation step can create a more expensive workflow.

Ask reviewers to distinguish a quick confirmation from a rewrite. If a user has to reopen every source and reconstruct the reasoning, the assistant may have accelerated drafting without reducing the burden of checking. Conversely, a service that reliably retrieves the right evidence may be valuable even when its prose needs modest editing.

The platform has to survive its first change

Before choosing, run one ordinary change through the proposed operating model: a source document is replaced, a connector fails or a model configuration changes. Can the team detect the effect, rerun the relevant evaluation and restore a working service? Identify which part is handled by the vendor and which needs your engineers.

This exposes a commercial distinction that demonstrations tend to hide. A platform with extensive flexibility may be the right choice for a capable development team. The same flexibility can become a permanent support obligation for a small team buying a ready-to-use service. Procurement should price that obligation alongside subscriptions and consumption.

A useful shortlist therefore has two stages. First remove options that cannot meet the workflow’s mandatory boundaries. Then compare the remaining candidates on accepted output, user effort, integration and sustained operating cost. Keep a limited pilot where a specific uncertainty remains; avoid extending it simply because nobody defined the buying decision.

The next step is to build the evaluation set for the intended workflow. Once that exists, suppliers can compete against the same work, and the enterprise AI lead can explain what the chosen platform will actually deliver.