Where the headroom is

Benchmarks matter here for two reasons that pull in opposite directions. They tell you where model capability is weak, which is where a buyer's budget goes. And they are the field's dominant marketing instrument, which means several of the most cited ones were built by companies selling the private version of the same thing.

Start with the capability picture, because it is unusually clear-cut.

#Enormous headroom in taste, none in compositionality

Read those together. On the objective, checkable properties — is there one red cube and two blue spheres — the benchmarks have been solved hard enough that they now mismeasure the models they were built for. On aesthetic discrimination, the best system in the world is closer to chance than to a professional.

A 42-point gap between the frontier model and the human expert is not a research curiosity. It is the budget. Everything on the buyer side of this dossier — The generative-media labs, The creative tool layer — flows from the fact that the capability gap in taste is enormous, measured by two independent groups with different task formats, and shows no sign of closing on its own.

Saturation in this field is not one curve. Three distinct things are happening:

1. Compositional and objective benchmarks are saturating and being replaced. GenEval → GenEval 2 is the clean case, with a quantified drift figure. VBench → VBench-2.0 is the same move in video.
2. Aesthetic benchmarks are nowhere near saturation. 26.5% against 68.9%; 54% against a 50% floor. There is a headroom abundance.
3. Arena-style leaderboards do not saturate; they drift. They are relative, so there is always a winner. That makes them durable marketing and weak products (The arena layer).

#The design and UI benchmarks

Two things to take from the table beyond the numbers. UI-Bench is the most rigorous public human-judged design benchmark in existence and it uses TrueSkill rather than Bradley-Terry specifically to get calibrated confidence intervals rather than point estimates — a methodological choice worth copying. And the two most useful design benchmarks disclose radically different things about their panels: VAB gives 10 expert judges per task and a controlled study design; UI-Bench names 194 experts with a role breakdown but states only that they were "hand selected and invited," thanking them for "generously contributing their time and judgment," which reads as unpaid (arXiv 2508.20410).

#The image and video benchmarks

• GenEval — compositional text-to-image, object-detection-based. Explicitly drifted. GenEval 2 documents benchmark drift of up to 17.7% absolute error against human judgement and introduces Soft-TIFA, decomposing into visual primitives (arXiv 2512.16853).
• T2I-CompBench / ++ — compositional attributes, relationships, complex compositions (NeurIPS 2023 D&B). Academic, open, largely saturated on the simpler splits [WEAK].
• GenAI-Bench — compositional text-to-visual generation with VQAScore-family automatic metrics; used as a transfer target by HPSv3++, which reports 76.3% (arXiv 2606.14657).
• HEIM — Stanford CRFM's Holistic Evaluation of Text-to-Image Models; 12 aspects including aesthetics, originality, toxicity and fairness. Academic, open, and the intellectual ancestor of every multi-axis rubric in this market.
• VBench / VBench-2.0 — Vchitect (arXiv 2503.21755). VBench-2.0 moves explicitly from "superficial faithfulness" to "intrinsic faithfulness": Human Fidelity, Controllability, Creativity, Physics, Commonsense. Extensive human annotation used to validate the automatic metrics — with no annotator count given.
• EvalCrafter — text-to-video across visual quality, content quality, motion quality and text-video alignment, aggregated and calibrated to human opinion.
• MJ-Bench — evaluates the judges, not the generators: alignment, safety, image quality, bias. Its key finding is directly monetisable for a critique-first vendor: natural-language Likert feedback from VLM judges is more accurate and stable than numerical scoring (arXiv 2407.04842). The written rationale beats the number.

#Which ones are vendor-captured

This is the finding that should change a positioning plan.

UI-Bench was built by AfterQuery — "a research lab curating high-quality, human-generated data for AI foundational models" (arXiv 2508.20410 author affiliations; StartupHub). That is the same play as Contra Labs' Human Creativity Benchmark: an expert-data company publishes an open, rigorous, human-judged benchmark in its own vertical, uses it to establish methodological authority, and sells the private version.

AfterQuery ran that play in the design vertical roughly a year before Contra Labs launched. Any positioning that treats "publish a design benchmark to establish credibility" as novel is a year late in this specific niche — see AfterQuery and UI-Bench and Publishing the benchmark.

Sorting the landscape by who owns the scorer:

I note without asserting causation that the top-ranked tool on UI-Bench, Orchids, holds the largest margin over the field. There is no evidence of any relationship between AfterQuery and Orchids and none is alleged. But the general point holds for the whole category: when the benchmark author is a commercial party in the same market, the benchmark is not neutral infrastructure, and buyers increasingly know it.

The counter-move is governance, and it is cheap: an independent methodological advisory board, published conflict-of-interest disclosures, and pre-registered evaluation protocols. No US vendor in this market currently offers any of the three, and it is exactly what a European institutional buyer responds to (The one right a US competitor cannot hold).

#What this means for a new entrant

Do not publish a benchmark to establish credibility. Publish a benchmark to establish a method. The credibility play is saturated; the methodological ground is not. The two open positions are the reliability-per-axis-and-phase work in Phase decomposition and the panel-sizing question in The oracle problem — neither of which any competitor has published, both of which are cheap, and both of which produce a number a buyer can use rather than a leaderboard they will forget.

And remember what a published benchmark costs you. Every published benchmark is a spent asset — the moment a de-novo brief set is public, it can leak into pre-training and its value as an uncontaminated test collapses. The resolution is the two-tier corpus in Eight units and one cost anchor: a public tranche sized for citation and inbound, and a permanently sealed tranche that is the actual product.