Irregular

Irregular runs cyber evaluations for OpenAI, Anthropic, Meta and Google DeepMind, and its SOLVE framework and CyScenarioBench appear directly in frontier model system cards (BERI). CyScenarioBench is named in the Claude Opus 5 system card with a disclosed 33.7% completion score (Anthropic).

#What $450M buys with 35 people

The valuation is not built on headcount or on data volume. It is built on being the named third party in the safety disclosure of the labs that compete with each other. A system card citation is a public, durable endorsement a competitor cannot purchase, and Irregular holds four of them across the firms with the largest evaluation budgets.

That is the same asset Gray Swan AI holds — cited in eleven system cards — and it is the reason the Offensive security vertical has the hardest demand evidence anywhere in the atlas. The Claude Opus 5 card alone names UK AISI, Irregular, an academic ExploitGym consortium, Mozilla, Trajectory Labs PBC, 10a Labs and Gray Swan for cyber evaluation (Anthropic). Nothing comparable exists for Design and UI/UX, Law or Accounting, audit and tax.

#The structural oddity nobody has priced

BERI's framing is the finding: three competing labs bought "independent" assurance from the same 35-person firm (BERI).

Two readings, both live.

The optimistic one is that shared vendors are how assurance markets normally work — audit, penetration testing and certification all converge on a handful of accepted names, and convergence is what makes results comparable across labs. Once your framework is the shared reference, switching costs are collective rather than individual, which is the strongest form of hold in the atlas.

The pessimistic one is that a 35-person firm is a single point of methodological failure for the public safety record of four frontier labs, and that concentration of this kind attracts either a regulator or an in-house replacement. Labs already run private suites — ExploitBench and ExploitGym sit alongside CyScenarioBench in the same card. The vendor-owned benchmark is a strong position until the buyer decides assurance should not be procured from one shop.

#What this does to the entry window

The decisive fact for anyone considering offensive security as a niche is that the benchmark layer here is already captured by vendors. Irregular owns SOLVE and CyScenarioBench; Gray Swan owns Arena and its million-plus attack trajectories; the labs own ExploitBench and ExploitGym. A new public benchmark — the standard opening move described in The specialist wedge — is a much weaker weapon in this domain than in one where no reference exists.

Note the inverse relationship this creates across the atlas. The niches with the most proven lab spending are the niches where the benchmark layer is already owned. The niches with an empty benchmark layer — defensive security, Law, Accounting, audit and tax — are empty precisely because the budget has not arrived yet. You are choosing between fighting for a proven budget and manufacturing one; Vals AI is the existence proof that manufacturing works.

Public academic benchmarks remain unsaturated but are not the procurement reference: NYU CTF Bench puts Claude 4.5 Opus at 59.0% (118/200) and Gemini 3 Pro at 52.0% (arXiv 2604.17159).

What to learn from it: the durable asset in evals is not the data or the model — it is being named in someone else's system card, and that citation is bought once and compounds, which is why the vendors who have it are valued on it rather than on revenue.