Miju LabsTwo capability curves are moving in opposite directions, and the gap between them is the commercial opportunity.
Offensive is saturating. OpenAI reports CTF performance moving from 27% to 76% in three months (OpenAI). Cybench's top score is 0.776 for GPT-5.3 Codex. Anthropic's Opus 4.6 system card states flatly that the model "has saturated all of our current cyber evaluations, achieving ~100% on Cybench (pass@30)," and that "[t]he saturation of our evaluation infrastructure means we can no longer use current benchmarks to track capability progression." By Opus 5, CyberGym is listed as retired.
Defensive is not close. CrowdStrike and Meta's CyberSOCEval puts frontier models at 23–34% on malware analysis and 43–53% on threat-intelligence reasoning, against random baselines of 0.63% and 1.7%, and states plainly that "current LLMs are far from saturating our evaluations" (arXiv 2509.20166). Microsoft's ExCyTIn-Bench tops out at 56.2% for GPT-5 at high reasoning (Microsoft). Microsoft's CTI-REALM tops out at 0.637 for Claude Opus 4.6 overall — and 0.282 on cloud multi-step attack tasks (arXiv 2603.13517).
The scoreboard
| Benchmark | Owner | Task | Best frontier score | Saturated? |
|---|---|---|---|---|
| CyberSOCEval (malware) | CrowdStrike + Meta | Malware analysis, 609 questions | 23–34% (random 0.63%) | No — wide open |
| CyberSOCEval (threat intel) | CrowdStrike + Meta | CTI reasoning, 588 Q / 45 reports | 43–53% (random 1.7%) | No |
| CTI-REALM-50 | Microsoft Security AI | End-to-end detection rule generation | 0.637 Opus 4.6; 0.282 cloud | No |
| ExCyTIn-Bench | Microsoft | Multi-step threat investigation | 56.2% GPT-5 High | No |
| AthenaBench | Academic | 6 CTI tasks | 66.1% combined; 39.0% attribution; 32.6% mitigation | No on reasoning tasks |
| CTIBench | Academic (NeurIPS'24) | 5 CTI tasks | GPT-4: MCQ 71.0%; attribution 52% correct / 86% plausible | Partially |
| DFIR-Metric | Academic | Forensics, 3 modules | GPT-4.1: MCQ 92.75%, CTF 28%, NIST 38.52% | MCQ saturated; practical wide open |
| AuditBench | Academic | Attack investigation, 51 scenarios | Lab F1 1.00; OpTC classification 0.45, lateral movement 0.25 | No on real data |
| Malcat LLM benchmark | Vendor practitioner | Malware triage + static unpacking | Triage 15.2/20; unpacking 11.8/20 | No |
| SSVC triage study | Academic | Vulnerability prioritisation, 384 CVEs | F1 0.79 exploitation; 0.43 mission/wellbeing | No |
| Cybench | Academic | Offensive CTF | 0.776 GPT-5.3 Codex | Approaching |
| MITRE ATT&CK Evaluations | MITRE | Product detection efficacy | — | Losing participants |
That last row matters more than its blank score. Palo Alto Networks, SentinelOne and Microsoft all withdrew from the 2026 MITRE ATT&CK Evaluations, announced around 22 September 2025, citing resource reallocation to product roadmaps (Cybersecurity News). The industry's flagship independent evaluation is losing its largest participants at exactly the moment vendors are publishing their own. Independent evaluation capacity is being vacated.
Reasoning models do not help, and that is the whole argument
In coding and mathematics, test-time reasoning is the reliable lever. In defensive security it is not, and two independent vendor-scale benchmarks say so.
CyberSOCEval reports that test-time reasoning models "do not achieve the boost they do in areas like coding and math, suggesting that these models have not been trained to reason about cybersecurity analysis" (arXiv 2509.20166). CTI-REALM, built by a different company on a different task, found medium reasoning effort outperformed both high and low, and that dedicated reasoning models (O3, O4-Mini) underperformed general-purpose ones (arXiv 2603.13517).
If more inference-time compute fixed defensive security, the gap would close on its own schedule and no data purchase would be needed. Two hyperscalers independently finding that it does not is the strongest available evidence that the bottleneck is training data, not model scale — and specifically that nobody has ever trained a model on a corpus of practitioners reasoning through defensive work, because no such corpus exists.
CTI-REALM sharpens it further. The most discriminating checkpoint in the whole benchmark was query execution, where Claude models scored 0.86–0.92 and most OpenAI models fell below 0.50; removing CTI-specific tools cost models 0.077–0.150 reward points; and seeding GPT-5-Mini with domain guidance closed only 33% of the gap. The paper's conclusion is that "query construction remains an inherent model capability limitation rather than a knowledge deficit." Knowledge injection does not fix it. Practice traces might.
The knowledge/practice split, and what it kills
DFIR-Metric is the sharpest single result in the literature. GPT-4.1 scores 92.75% on 700 certification-grade forensics multiple-choice questions, 28% on practical CTF-style challenges, and 38.52% on 500 disk-analysis tasks adapted from the NIST Computer Forensics Tool Testing Program. The authors conclude that "practical digital forensic scenarios and end-to-end forensic workflows remain out of reach for current models" (arXiv 2505.19973).
AuditBench shows the same shape in a different dimension — clean data versus dirty. Across 51 scenarios (25 lab-generated on Linux and Windows VMs via Atomic Red Team, 26 curated from DARPA's OpTC dataset), models reach F1 1.00 on lab classification and lateral movement and collapse to 0.45 and 0.25 on real OpTC data, with the paper noting that "LLMs frequently reported a wide spectrum of benign activity as instances of attacks" (arXiv 2606.10281).
AthenaBench repeats it a third time: 92.0% on CTI knowledge, 39.0% on threat actor attribution (arXiv 2511.01144). Four independent research groups, the same pattern.
Multiple-choice security knowledge is saturated. Any dataset product shaped like questions and answers is selling into a solved problem — the Q&A market is over, and the certification-style corpora that used to be the obvious first product are now worth roughly nothing to a frontier lab.
What is not saturated is doing the work: the trajectory, the intermediate reasoning, the rejected hypothesis, the benign-discrimination call. Trajectories and artefacts, not questions and answers. That is the same conclusion The oracle decides everything reaches from the oracle side, arrived at from the measurement side.
There is a second-order finding hiding in AuditBench's collapse. Lab data teaches attack recognition; it does not teach false-positive suppression, which is where the commercial value in a SOC actually sits. That is the structural reason SOC alert triage resists productisation as data, and it is covered in Defensive security.
Every defensive benchmark was built synthetically, and says so
This is the awkward structural fact, and it is stated by the benchmark authors themselves, in their own limitations sections.
- CyberSOCEval generated its questions with Llama 3.2 90B and had them "manually validated and edited by cybersecurity experts" — without stating how many experts, how much time, or what it cost, and conceding "it remains the case that this is a potential source of bias."
- CTIBench used GPT-4o to prepare the MCQs, with manual quality analysis afterwards (arXiv 2406.07599).
- AthenaBench used GPT-generated realistic scenarios for two of its six tasks.
- The SSVC triage study had to invent synthetic organisational stand-ins for the "Mission & Wellbeing" decision point, because no ground truth existed — and that is precisely the decision point where models scored worst, at F1 0.43 (arXiv 2510.18508).
No published defensive benchmark has been built primarily from expert-generated ground truth. Every one is synthetic generation with expert review bolted on, because expert generation was too slow.
The cost of the alternative is documented exactly once. CTI-HAL, a human-annotated CTI dataset, used two independent annotators on 81 reports over eight weeks, reaching Krippendorff's alpha of 0.70 (arXiv 2504.05866). That is roughly five reports per annotator-week. At the $80–95/hour rates in Defensive supply, five reports a week per person is the throughput number any expert-generated corpus has to be priced against — and it explains, in one figure, why every benchmark author chose the language model instead.
That is the opening and the warning in the same sentence. The benchmark authors are describing the product in their limitations sections; they are also telling you what it will cost to build.
I searched specifically for an LLM benchmark on cloud misconfiguration review — infrastructure-as-code in, misconfiguration plus exploitability reasoning plus blast radius out — and found only vendor CSPM tool comparisons. No academic or vendor benchmark surfaced [COULD NOT ESTABLISH].
That is either the widest unmeasured gap in defensive security or a search failure, and the two are not distinguishable from here. The one adjacent hard number points at a genuine gap: CTI-REALM scores cloud multi-step attack detection at 0.282 against 0.585 for Linux tasks, concluding that "current frontier models struggle with the correlation and contextualization demands of advanced cloud threat detection."
If it really does not exist, the move is to build the benchmark first and sell the data second — defining the metric is the more durable position, which is the argument in Publishing the benchmark.
Who owns the good ones
Worth being precise, because it changes who a specialist competes with.
Genuinely public and independent: CTIBench, AthenaBench, DFIR-Metric, AuditBench, SecBench, Cybench, AutoPenBench. All academic, all open — and none built by a party that holds the telemetry.
Vendor-built but openly published: CyberSOCEval (CrowdStrike + Meta), CTI-REALM and ExCyTIn-Bench (Microsoft), CyberSecEval 4 (Meta). These are the highest-quality defensive benchmarks in existence and they exist because vendors donated substrate academics could not get. Microsoft is routing CTI-REALM into the UK AISI's inspect_evals framework, which makes AISI a named institutional home for defensive evaluations — see Government buyers.
Vendor-internal, never published: CrowdStrike's telemetry-grounded evaluations, asserted in its blog and not released; every AI SOC startup's internal eval set; Microsoft's production Defender grading set of 1,088 alert-level grades from 208 customer organisations, described in a paper and not shared.
CrowdStrike has staked out the position that meaningful evaluation requires "real telemetry from actual intrusions (not synthetic data)" and access to "trillions of events each day," while criticising public benchmarks for "agenda capture," saturation and contamination (CrowdStrike). That is a competitive threat and a validation of the thesis in the same paragraph — and on SOC triage specifically, AuditBench suggests they are right.
The gap is therefore not that no benchmarks exist. It is that the good defensive benchmarks are vendor-donated, small (typically 50–700 items), synthetically generated with light expert review, and none of them measures the false-positive discrimination that determines real-world value. The measurement stack in Offensive security is exhausted; the one in defensive was never built properly in the first place.