Background
Patwardhan works on OpenAI's Preparedness and evaluations line and has been an author on a run of expert-graded benchmarks — the pattern across her work is recruiting genuine domain experts rather than crowdworkers and having them both author and grade tasks. GDPval (arXiv 2510.04374, October 2025; accepted at ICLR 2026, senior author Jerry Tworek) is the fullest expression: tasks grounded in the Bureau of Labor Statistics Work Activities framework, deliverables that look like real outputs rather than exam questions, and a public automated grading service at evals.openai.com to scale the comparison.
The headline finding — frontier models approaching industry experts in deliverable quality — matters less to Julian than the machinery. GDPval is a procurement template: identify occupations, recruit practitioners with a decade-plus of experience, extract representative deliverables, build a grading protocol, then automate. Creative occupations sit inside the GDP sectors GDPval covers.
What they run now
Evaluations that measure real economic capability rather than exam performance: GDPval and its automated grader, plus Preparedness-track evaluations. The operational core is expert recruitment and grading protocol design — who counts as an expert, what a representative deliverable is, how two experts are made to agree.
What they buy
Direct and evidenced in kind, if not by vendor name. GDPval required sourcing experienced practitioners across 44 occupations and collecting both deliverables and expert grades — that is a large human-data procurement, whether run in-house or through vendors. For Julian the specific opening is occupational coverage: graphic designers, art directors, photographers, video editors, UI/UX designers and marketing creatives are all BLS occupations inside GDPval's sectors, and creative deliverables are the hardest category to grade automatically. A vendor that can supply senior creative practitioners plus a defensible grading protocol is selling exactly what GDPval consumes.
Career
- presentResearcher — Preparedness / evaluations, OpenAI — Lead author GDPval; co-author on expert-graded benchmark work
- —Harvard University, AB