GDPx tests agents against demanding specialist workflows GDPx tests agents against demanding specialist workflows

Different work, different acceptance rules

GDPx selects domains that SemiAnalysis expects to benefit most from frontier AI, including chip design, biology and software engineering. Tasks normally allow eight hours, with shorter limits for some. Models use their native agent harness when available and OpenCode otherwise. The headline averages three attempts per task using each task’s intended binary or continuous grading. Domains receive equal weight, as do partners within each domain. These choices make the result a measure of the tested model-and-harness combination.

The original occupational benchmark asks another question

OpenAI introduced GDPval on September 25, 2025, with 1,320 tasks spanning 44 occupations in nine economically significant industries. Its public gold subset contains 220 tasks. Experienced professionals compare model deliverables with human-produced work, without knowing which is which, and judge whether the model’s output is better, equivalent or worse. The initial evaluation is one-shot. OpenAI explicitly identifies missing iteration, client feedback and ambiguous task discovery as limitations. A finished document can therefore be the unit of comparison without representing an entire working relationship.

A smaller score can come from a different test

The comparison changes both the work sampled and the meaning of success. A preference judgment asks which deliverable a reviewer would choose. An acceptance test can reject an otherwise useful artifact for failing one required condition. Neither ordering is inherently wrong: a draft for human revision and an unattended production handoff have different buyers. Comparing their percentages directly would discard that distinction. The relevant question is whether a benchmark’s rejection rule resembles the failure that would stop the intended deployment.

An apparently complete workbook can still fail

The launch report describes Living Context’s finance task, which requires completing a quality-of-earnings workbook while reconciling supporting records. None of the 25 selected attempts passed; the strongest individual attempt met eight of eleven checks. Reported errors include treating an email or schedule as more authoritative than an executed agreement. That is a failure of evidence selection, not simply calculation. A spreadsheet can remain internally consistent after the wrong source supplies its assumptions. More polished formatting would not repair that dependency.

Public traces expose the work behind the score

The rollout site publishes examples for one task per partner and keeps the remaining tasks private to discourage repeated tuning against them. Its launch collection offers 168 transcripts. Examples range from completing an unfinished acquisition workbook to modifying a startup’s financing charter and designing a compressor housing. The page pairs task descriptions with recorded actions and run outcomes. These records make it possible to inspect how an agent approached a problem, rather than inferring its behavior from a model name. They also separate evidence of an attempted workflow from a screenshot of its final artifact.

Acceptance rules belong in the purchasing decision

Equal domain weighting gives a specialist field the same influence regardless of its share of a particular customer’s workload. A company buying a document assistant and one buying a chip-design agent therefore need different slices of the result. The strongest aggregate entry is not automatically the most relevant candidate for either deployment. Cost also needs a denominator: spending less on an attempt is different from spending less on an accepted result. Procurement based on successful handoffs would reward dependable completion, while a drafting assistant might justify its cost through useful partial work.