Application security metrics: explain what changed before celebrating the count

The application security team rolls out a scanner to more repositories and reports twice as many findings. The next month, its total falls sharply after several projects stop uploading results. A simple trend chart would make the first month look worse and the second look better.

Both changes could be measurement effects. For an application security lead, the job is to explain whether relevant exposure changed and what engineering decision follows. A finding count helps describe the workload. It cannot carry that conclusion on its own.

In a hypothetical programme, ten applications initially produce 200 open findings. Adding ten previously untested applications reveals another 150, with no change in the original applications. The total rises to 350: a 75% increase in findings alongside a doubling of application coverage. That does not demonstrate a 75% deterioration in software security. It demonstrates that the organisation can now see more issues requiring assessment.

Reverse the situation. If the new applications cease reporting and the dashboard excludes their stale results, the count returns to 200 without a single fix. The apparent 43% reduction is lost visibility. The reporting population must remain visible beside the total, including systems whose checks have stopped or become incomplete.

Explain the movement, then assess the exposure

Reconcile the opening and closing population of findings. Separate newly detected issues, verified remediations, duplicates, accepted exceptions, rule changes and assets entering or leaving scope. If the product’s dashboard uses different state definitions, map them before comparing periods.

Preserve a comparable view of applications covered in both periods, while showing newly covered services separately. Neither view is sufficient alone: the stable group helps assess change, and the new group shows risk the programme previously missed. Include the age of the last successful relevant analysis so dormant checks cannot be mistaken for clean results.

Now examine which unresolved issues matter to the service. Consider the deployed version, affected functionality, accessibility of the vulnerable path, existing controls and business consequences. In NIST’s Secure Software Development Framework 1.1, the vulnerability-response practices call for analysis sufficient to plan a risk response, including exploitability and potential impact, and for addressing root causes. That supports a management discussion about exposure and recurrence, beyond the volume processed by the testing tools.

Do not make confirmed reachability the only reason to act. An incomplete analysis can leave reachability unknown, and a configuration change can expose a previously constrained path. Keep unknowns visible and decide what investigation or precaution the consequence warrants.

A “fixed” alert needs a deployment boundary

Even the status label can hide an important distinction. GitHub’s code-scanning documentation explains that an alert page reflects its status on the default branch. An alert listed as open on another branch can therefore appear fixed on the detail page if it has already been fixed on the default branch; the affected-branches section shows the other branch’s status.

For an AppSec lead, the practical question is which revision is actually deployed. A fix on the default branch may still await a release, while a supported older version continues to carry the issue. Report code remediation and deployed remediation separately where that delay affects exposure.

The same discipline applies to closure by dismissal or acceptance. A valid false-positive decision can reduce wasted engineering effort, but it is not a vulnerability fix. An approved exception can make a risk decision accountable, but it does not make the underlying condition disappear. Combining these outcomes into one closure rate obscures what the team achieved.

Use the report to change the work

Pair the volume trend with the age of consequential open issues, time to verified deployment of fixes and recurring defect patterns. Show the slow unresolved cases as well as averages for completed work. A mean calculated only from closed tickets can improve while the oldest, hardest issue remains untouched.

Ask what intervention the evidence supports. Repeated authorisation defects may justify engineering changes to a shared component and focused review of its consumers. A long interval between fix and deployment points towards release constraints. Missing analysis on critical applications calls for restoring coverage before buying another tool.

The programme should reward useful discovery and durable correction. Suppressing difficult findings can improve the chart while weakening the service. Wider coverage can worsen the chart while giving the organisation its first credible view of the work ahead.

Where coverage is the unresolved problem, use SAST, DAST and SCA evaluation around the delivery workflow to identify the evidence another testing method would add. The next purchase should answer a gap in understanding or action, rather than produce a more flattering total.