OpenAI’s wet-lab benchmark separates credit from correctness OpenAI’s wet-lab benchmark separates credit from correctness

What does the percentage measure?

OpenAI reports that GPT-6 Astra earned 66.7% of available credit at xhigh reasoning effort. The grading assigns credit to individual criteria. That number describes the average task score, not a count of fully correct experiments. A percentage has no useful operational meaning until its denominator is clear. An evaluation can count correct choices, completed tasks or points awarded across several parts of an answer. Those measurements can all be legitimate while supporting different conclusions about the same system.

How can partial credit hide a decisive error?

Consider a hypothetical evaluation with three equally weighted criteria per task: identify the evidence, calculate a value and select the final action. An agent that gets the first two right on every task scores two-thirds of available points. It still gets every final action wrong. Another agent could earn all three points on two tasks and none on a third. Its average would be identical, despite completing two of those three tasks correctly. This is an illustrative scoring example, not a reconstruction of BioBasicsBench. It shows why an average cannot reveal where the errors occur. The weighting expresses what the evaluator values; it does not automatically express the cost of acting on a mistake.

How does an older biology benchmark differ?

FutureHouse introduced LAB-Bench on July 16, 2024, with 2,457 questions across eight categories. Its tasks cover research activities including literature retrieval, database use, figures and tables. The format is multiple choice. The authors explicitly discuss a weakness: eliminating implausible options can improve results without demonstrating that the model can independently derive the answer. They recommend validating such performance through manually graded open answers. They also report an important tooling difference in their initial comparison: human participants could use tools, while the evaluated models did not. These are features of that historical study, not a ranking of current systems.

Why can two benchmark scores disagree?

A multiple-choice task supplies candidate conclusions. An open response requires the system to produce its own. A rubric can reveal which parts of that response were useful, while an exact-match score compresses them into a single decision. Neither format supplies a universal exchange rate between percentages. A ten-point increase on one test cannot be added to, or substituted for, ten points on another. The useful comparison starts with the action being measured and the assistance available during the test.

What would make the result useful in a product?

For a software team, the missing link is the consequence of each output. A tool that suggests an annotation has a different decision boundary from one that authorizes the next step. In our worked example, recording final-action correctness separately immediately distinguishes the two agents. Keeping the intermediate scores preserves information about useful progress. That produces two interpretable measurements instead of asking a single average to describe both assistance and autonomy.