Anthropic explains why safety scores need deployment context Anthropic explains why safety scores need deployment context

What changes when the settings change?

Anthropic reports an API-versus-claude.ai gap of up to 34 percentage points in Fable 5.1 testing. Its guide calls for documenting model settings and distinguishing observed differences from demonstrated causes. The tested configuration is part of the result.

Is this only a concern raised by a model vendor?

The BetterBench paper, submitted on November 20, 2024, assessed 24 benchmarks against 46 practices covering their lifecycle. Its authors found substantial differences in quality. Most benchmarks in that assessment did not report statistical significance or make reproduction straightforward. The study supplied a minimum-quality checklist and a repository of assessments. That earlier work examines the measurement infrastructure itself: the evidence behind a ranking and its first-place entry. It provides independent context for the new guide without independently verifying Anthropic’s model results.

How can a test follow an entire interaction?

Petri, introduced on October 6, 2025, uses an auditor agent to interact with a target model through simulated users and tools. A separate judging step scores transcripts. Researchers supply initial instructions describing the behavior they want to investigate. The auditor can adapt as the target responds, instead of following one fixed script. Anthropic’s original pilot covered 14 models and 111 starting scenarios. Its release explicitly presented those metrics as provisional and encouraged reading the underlying transcripts. The pilot’s interactions take place in constructed evaluations with simulated users and tools.

What would the distinction mean in practice?

Consider a hypothetical support assistant tested on 100 invented conversations. In one configuration it drafts replies but cannot send them. In another it can send messages to customers. Suppose both receive 95 passing grades under a rubric that checks only the words in the final answer. Those equal totals do not establish equal behavior. The second configuration could have sent an incorrect message before producing an acceptable final reply. These invented numbers serve only to illustrate the distinction. Adding a check for sent messages changes what a passing grade means. Reusing the old percentage after changing that definition would compare two different tests. Keeping the conversation record and action record separate makes the discrepancy visible: one records what the assistant said, the other what it did.

Where does the evidence stop?

A deployment decision in that example concerns the sending assistant and its permissions. The useful comparison would hold the invented cases constant while recording both answers and actions. A passing final response cannot undo an earlier send. Equally, a failed case identifies a specific failure under those conditions; it does not prove every conversation will fail. This is why the release question needs a named configuration and a defined failure, alongside its aggregate number.