AI SOC analyst evaluation: count the correction work as well as the speed

The assistant writes a clear incident summary in seconds. It names an endpoint, explains a likely sequence and proposes a response. Then an analyst discovers that the endpoint belongs to a different event. The prose was useful-looking; the decision support was wrong.

For a SOC manager, that hypothetical case is a better starting point for an AI evaluation than a polished demonstration. The product must improve a defined task using the evidence available to the team. Fluency, detection quality and authority to act are separate things to assess.

Identify the work the feature actually does

Microsoft describes Security Copilot as supporting tasks including incident response, hunting and posture management, with plugins supplying context from security products and other sources. That breadth is useful context for buyers, but each workflow needs its own acceptance evidence.

An assistant that summarises an incident is not necessarily detecting a threat that existing systems missed. A feature that drafts a query is not necessarily executing it correctly. An agent that can change an account introduces a different consequence from either of those assistive functions.

Define the pilot around one task: for example, preparing a triage recommendation from a bounded set of incident evidence. Compare the result with how analysts perform that task today, including the time spent collecting context and checking conclusions.

Build cases that expose plausible mistakes

Include ordinary cases, ambiguous cases and cases with missing information. A useful test set should contain situations where the right outcome is to request more evidence rather than confidently choose a response. Keep the expected outcome and rationale available to the reviewers.

Check whether the system associates the correct identities, devices and timestamps. Ask whether cited records actually support the claim. A link can be valid while the conclusion drawn from it is wrong.

Keep the same evidence boundary for the baseline and assisted runs. If one group sees additional context or receives help from the supplier, the comparison cannot isolate the effect of the feature. Rotate comparable cases or participants where practical to reduce familiarity effects.

Measure the whole task

Use an explicitly hypothetical example. An analyst spends 12 minutes on a case without the assistant. The assisted workflow takes three minutes to generate and assemble the recommendation, four minutes to verify it and two minutes to correct it: nine minutes in total. That is a three-minute saving for that case, not a nine-minute saving based on generation time alone.

Across a pilot, include failed cases and additional escalations. Separate harmless wording changes from errors that would alter the response. A single average can hide a small group of severe mistakes, so report the categories and the resulting reviewer workload.

Do not translate time saved automatically into headcount savings. Analysts may use the released capacity to investigate more cases or improve evidence quality. That can be valuable, but it is a different business outcome and needs to be described accordingly.

Test permissions separately from recommendations

If the feature can act, verify where approval and permission checks are enforced. The relevant question is whether the executed action stays within the approved boundary, including after retries or a changed plan.

Begin with a bounded workflow and expand only when the additional authority has its own evidence. The agent evaluation guide examines partial failure and tool behaviour; these issues should not be hidden inside a general satisfaction score.

Make the pilot capable of saying no

Set acceptance criteria before the supplier demonstration. They should cover useful accuracy, correction effort, unresolved cases and operating cost. If a feature saves drafting time but increases the burden of checking, the team should be able to reject or narrow it.

Use a representative evaluation set and retain enough case-level evidence to explain the result. The buying case becomes credible when the SOC can show which work improves, what remains human-led and where the feature should not yet be used.