Skip to content
← Back to Computer science

EU agencies had AI models audit 27 codebases: of 88 findings the owners judged so far, 95% were real

ENISA and CERT-EU's joint report says the models also overrated severity, and that checking the 220 findings is the slow part.

95%of 88 assessed findings were true positives (220 found)

ENISA, the EU’s cybersecurity agency, and CERT-EU, the security team for the EU institutions, report an experiment in which frontier AI models read the source code of 27 codebases used by the EU institutions, 25 of them already open source and 2 about to be. The scans ran in August 2026. The models were OpenAI’s GPT-5.5 and GPT-5.6 families (including Cyber versions) and an Astra model, reached through its Daybreak programme, plus Anthropic’s Mythos 5 through Project Glasswing. They ran in tools including OpenAI’s Codex Security plug-in and Vigilo, CERT-EU’s own harness. The models saw only code, with no deployment information, and no live system was tested.

They produced 220 potential issues. By publication the owners had assessed 88 (40%) and confirmed 95% as real. The rest were false positives or did not apply, mostly because runtime information had been withheld. Four new CVE identifiers (public IDs for security flaws) went to one of the 27 codebases. Owners rated the 88 at 26% high severity, 34% medium, 29% low and 11% informational.

General-purpose coding agents with a security plug-in were usually faster and cheaper. Vigilo, which adds an adversarial AI pass over each finding, took longer and used more tokens, and owners kept its severity rating in 74% of cases against 60% for the general agents. Most changes were downgrades. The report gives no results per model. Its rough cost for all scans is EUR 6,649 (about EUR 132 per 10,000 lines of code), though access was free.

What it means

This is a public body, not a lab, reporting how AI code audits went on software it is responsible for. Its main finding concerns people: the models produced findings faster than maintainers could check them. The agencies suggest tools that merge duplicates and triage findings before owners see them.

Caveats

  • The 95% covers only the 88 findings owners had assessed. The other 132 were unassessed, and owners who answered first may not be typical. The report does not say how many true positives each model or tool produced.
  • Severity “before” is the AI’s own rating. The report says the models leaned toward overrating, and that without deployment context severity is rarely knowable from source code.
  • ENISA built proofs of concept for “a limited number” of codebases only. Elsewhere the result rests on the maintainers’ word.
  • The report calls the cost figure indicative. The models were those ENISA had been granted access to, not a head-to-head selection.
  • OpenAI’s post of 17 November also lists these results. It says “projects” where the report says “codebases”; the figures match.
Suggest a correction