Ai2 tests how language models follow changing state in long contexts Ai2 tests how language models follow changing state in long contexts

Retrieval is only the easiest case

The four tiers move from immutable assignments through replacement values and self-referential updates to dependencies between variables. Ai2 tested Gemini 3.8 Flash, Claude Opus 5.5, and GPT-6 Sol and Luna. The researchers report strong results on the first two tiers, followed by sharp deterioration on harder, longer sequences. Accuracy reached 13% or less on the longest general-finite problems. Separately, recurrent and hybrid models trained directly on these tasks generalized farther than transformers. That experiment distinguishes learning a task from having acquired the capability during general pretraining.

The scoring protocol changes the comparison

The paper reports open pretrained models’ best result across six surface formats, rather than their average. Its frontier evaluation asks for the complete final state instead of selecting among five answers. Those protocols measure different response requirements. The researchers also analyzed CruxEval Python programs by asking how many execution steps could change the output if a variable changed. Those programs contained both high- and low-sensitivity cases, connecting the synthetic hierarchy to variation in actual code. Sorting training examples by increasing sensitivity barely affected Gated DeltaNet, hybrid, or position-free transformer models. It produced only a weak benefit for rotary-position transformers within the training distribution. A simple easy-to-hard curriculum therefore did not remove the measured difficulty. In that analysis, loops were unrolled along the path taken for a particular input. Sensitivity describes the consequences of changing intermediate values, not simply the number of tokens in a prompt.

Hold the underlying problem still

The generator’s identifiers let experimenters join corresponding examples instead of comparing unrelated samples. Copy operations read a variable’s current value; they do not create a persistent alias. The repository separately controls non-operative filler and the number of real updates. Inserting filler leaves the underlying trace and answer unchanged, allowing input length to grow without adding state transitions. For multiple-choice output, incorrect answers preserve the correct answer’s pattern of equal values. That blocks an answer-selection shortcut based solely on which variables match. Configured train/test separation checks semantic identifiers, then deterministically resamples collisions. A later split cannot be built before the split it must exclude. These controls make prompt appearance and semantic duplication explicit experimental choices. Configuration validation rejects unknown keys and insufficient value vocabularies before generation. Token-length bands are measured with a specified tokenizer, and empty bands are omitted. A formatter therefore need not supply every nominal context-length split.