IAM platform evaluation: make the difficult application part of the demonstration

The most revealing application in an IAM demonstration is often the one the supplier would prefer to leave until later. It may have old account conventions, local permissions or a business process that cannot tolerate an interrupted login. For an identity procurement lead, that is precisely why it belongs in the evaluation.

A smooth demonstration of the easiest integration establishes a starting point. Selection requires evidence about the access journey the organisation actually needs to support, including role changes, removal, failure and recovery.

Test the supported mechanism against your account model

Microsoft Entra documents application provisioning through SCIM and application-specific mappings. A standards-based connection provides a mechanism; the evaluator must still establish whether the application outcome meets the requirement.

Use existing test accounts with representative history. Check whether they match correctly, retain the intended resources and receive the right changes. A new account created in a clean demonstration environment avoids several of the problems that a real migration must solve.

Have the application owner judge the result with the identity team. The owner can confirm whether a permission actually supports the intended business task, while the identity team examines the event, mapping and failure handling. Neither view is sufficient alone.

Price the exceptions that survive selection

A platform can be a good choice without automating every application. What matters is whether the retained manual work is understood and supportable. Count the exceptions, their frequency and the specialist effort they require.

A cheaper licence may leave a larger integration or operating commitment. Conversely, paying for a broad feature set may make little sense if the required workflows use only a small part of it. Compare the complete service the team will operate, including transition and exit.

Build the evaluation around your estate

Inventory the applications that matter, the identity sources they depend on and the user groups they serve. Include contractors, privileged users and workloads where relevant. Record applications that cannot support the preferred integration method, rather than leaving them outside the demonstration.

Choose representative scenarios with the application owners. A leaver event, a role change, a failed provisioning update and an identity outage reveal different capabilities. Each scenario needs an expected result, acceptable timing and evidence from the affected application.

Set mandatory gates first

The NCSC identity guidance provides a foundation across access policy, identity management, privilege and operations. Translate those concerns into the application and lifecycle scenarios the shortlisted suppliers must demonstrate.

Separate requirements that are essential from preferences that can be traded off. A platform failing a mandatory application integration should not recover its position by collecting points for unrelated features. Mark unknowns as unresolved, not as passes.

Evaluation area	Demonstration to request	Evidence to retain
Lifecycle	Move a test user between roles and remove obsolete access.	Source event, platform log and resulting application permissions.
Integration	Connect a representative difficult application.	Supported method, dependencies and tested limitations.
Privilege	Grant and revoke a bounded administrative permission.	Approval, duration, activity record and revocation result.
Recovery	Exercise the agreed identity-outage scenario safely.	Steps, authority, dependencies and restoration result.
Migration	Move a small representative cohort with a rollback plan.	Coexistence behaviour, user impact and rollback evidence.

Use a common test pack

Give each supplier the same scenarios, datasets and acceptance criteria. Keep confidential production information out of an uncontrolled demonstration. Where a supplier needs a different approach, record the difference and assess its operating consequences rather than silently changing the test.

For an illustrative role-change test, create a user with access to one business function, move the user to another and inspect both sets of permissions. The test succeeds only when the agreed new access is present and obsolete access is removed or explicitly approved as a time-limited exception.

Capture failures as well as success. Ask what happens when an application rejects an update, a connector is unavailable or an approval is delayed. The operating team needs a visible failure and an actionable recovery route, not just a reassuring green status in the central console.

Score evidence, not confidence

Keep mandatory gates separate from weighted criteria. For each scored criterion, define what the score means before viewing responses. Retain the evidence and limitation behind the number so the selection team can challenge it.

The scorecard’s proposed scale runs from zero to four: demonstrated failure; major gap; partial fit; requirement met; requirement met with a specifically valuable additional benefit. Leave untested items blank. Adjust the definitions to your requirements before use.

Choose weights as a team and make their total visible. A high weighting for integration is defensible if difficult integrations dominate your operating risk; it is not a universal recommendation for every buyer. Test whether changing a disputed weight changes the shortlist, then resolve the underlying disagreement.

Include the transition and the exit

Ask how the platform will coexist with the current identity service, which applications move first and what would trigger rollback. Identify the support capacity required during enrolment and migration. A workable end state does not establish that the route to it is acceptable.

Also record what can be exported, how integrations would be dismantled and which controls need replacing at exit. Verify current supplier documentation and contractual terms for these questions.

Before selection, give the operations team the proposed runbooks and support boundaries. If the team cannot explain how it would handle a failed update or urgent access issue, the evaluation has not yet established operational readiness.

Keep a concise decision record with the requirements, gate results, scored evidence, operating costs, migration assumptions and reasons for the preferred option. Assign owners and dates to unresolved questions. The scorecard supports that judgment; it cannot make the decision on the organisation’s behalf.

For the scoring mechanics, use the software evaluation scorecard guide. Then carry the difficult application into the migration plan. The evaluation has done its job when the operating team recognises the service it is about to inherit.