Software evaluation scorecards: when the highest score should not win

Supplier A scores 82 and supplier B scores 78. The recommendation appears straightforward until someone asks whether A can export the records the business needs to retain when the contract ends. Nobody tested it.

For the procurement lead, the problem is not arithmetic. The score has answered a preference question while hiding an unresolved condition of purchase. A useful software selection scorecard separates those decisions before the demonstrations begin.

This is a pre-purchase comparison, rather than a supplier-performance report after contract award. Its job is to explain why an option is eligible, what evidence supports the preference and which assumptions could change the recommendation.

First establish eligibility

A mandatory gate needs a required outcome and an acceptable way to demonstrate it. “Supports enterprise identity” is too broad. “A departing employee loses the required application access through our agreed identity process” gives the technical team something meaningful to examine.

Microsoft Entra’s application provisioning documentation shows why the detail matters. Its provisioning service uses an application’s SCIM endpoint to create, update and remove identities; behaviour depends on mappings, scope and application support. A successful login demonstration does not establish the complete lifecycle integration.

The procurement consequence is a separate acceptance test for the capability the business requires. A supplier can show an attractive interface and still leave an unresolved removal or integration path. The IAM evaluation needs to establish that path before a general product score can support the purchase.

Give each mandatory condition one of three states: passed, failed or unresolved. Only passed options proceed to a recommendation based on weighted preferences. If no option passes, the next decision is to resolve evidence or reconsider the requirement openly. Do not silently turn it into a low-weight preference so the spreadsheet produces a winner.

Then make the trade-offs visible

The following hypothetical example uses a five-point preference scale. Scores describe the buyer’s assessment of demonstrated fit, not a market ranking. Weighted points equal the criterion weight multiplied by the score, divided by five.

Preference	Weight	Supplier A score	Supplier B score
Fit to the main workflow	40%	5	4
Implementation effort	30%	2	4
User experience	20%	4	3
Support fit	10%	3	4
Weighted total	100%	74/100	76/100

B wins narrowly because the team values implementation effort. Move ten percentage points from that criterion to workflow fit, making the weights 50%, 20%, 20% and 10%, and A reaches 80 while B remains at 76. Neither calculation is wrong. They represent different judgements about what matters.

That sensitivity is useful information for the sponsor. Before signing the recommendation, resolve whether the organisation is prepared to accept A’s greater implementation burden for its stronger workflow fit. A two-point lead should not be presented as decisive when a plausible shift in priorities reverses it.

Give a score an evidence rule

Define the meaning of the scale for each criterion before suppliers present. For workflow fit, a five might mean the representative task succeeds under the required conditions without a workaround. A three might mean it succeeds with a specified, acceptable manual step. The middle of the scale should describe a real outcome.

Keep “not tested” outside that scale. Awarding three points because nobody knows the answer turns missing evidence into apparent adequacy. Record the gap and its effect on the recommendation instead. Roadmap promises also need to remain distinct from demonstrated capability available in the proposed purchase.

Separate evidence strength from the score itself. A supplier assertion, a document and an observed test support different levels of confidence. If two candidates are close, the next test should target the uncertain criterion that could change the decision, rather than adding another broad demonstration.

Check for duplicate preferences as well. Scoring the same interface advantage under usability, adoption and employee experience can give it three votes. Keep criteria distinct enough that the weights express the sponsor’s priorities honestly.

Use the scorecard to shape the buying process

Apply the same scenarios and evaluation conditions across suppliers. Note where implementation assumptions differ: a managed integration and a custom connector may achieve similar outcomes while carrying very different delivery obligations.

Once those distinctions are explicit, convert the scorecard into the RFP’s requirements and acceptance tests. The recommendation can then explain eligibility, the preferred trade-off and the work still needed before commitment. The score remains useful because the buying team can show what sits behind it.