Local Qwen test identifies games from network weights Local Qwen test identifies games from network weights

Local reading test

Willison ran a quantized model through LM Studio using the researchers’ four game prompts. Apart from their weight values, he found those prompts identical. His model recognized Pac-Man, Pong and Breakout, while mistaking Freeway for a Pac-Man-like game. The successful runs took roughly forty-six minutes, sixty-five minutes and fifty-two minutes respectively. He increased the context allocation and request timeout to accommodate the work.

Reading and editing used different procedures

The original November 2 research blog describes eight main experiments. For the game test, GPT-6 Astra received numerical weights and a description of the grid and actions, without tools or network execution. The policies shared a 17,253-weight architecture. Other experiments allowed different forms of assistance. Editing Qwen3.5-0.8B involved a Python kernel, practice mathematics problems and up to fifty batched forward passes. A pass could generate answers for up to 512 practice problems. The authors attribute the resulting improvement largely to reducing repetitive reasoning loops. They explicitly distinguish that repair from adding mathematical capability. Their post labels the larger-model section as involving tool use. A follow-up on Gemma 3 270M, which rarely looped, did not establish a mathematics gain.

The criticism concerns what the experiments establish

In a November 3 response, spaCy creator Matthew Honnibal argued that presenting these results as one capability encourages readers to combine unlike evidence. Inspecting raw numbers and conducting code-assisted experiments pose different problems. His objection is to the broader inference, rather than a claim that the engineering work has no value. He acknowledges that the assisted editing itself represents useful work. A unified method that could both interpret and improve networks would support a stronger conclusion than a collection of procedures with different access to tools and feedback.

A rebuild adds evidence with a smaller baseline

A separately published Hugging Face checkpoint documents a November 2 rebuild by akhaliq using an AI assistant called ML Intern. It reconstructs the released edit and checks tensor hashes against the authors’ checkpoint. The rebuilt package contains a full multimodal model instead of the original text-only export; its language-model tensors match. Its evaluation reports 51 percent accuracy across five hundred MATH500 problems, sampled four times each. The comparison run for the unedited model covers only twenty-five problems. That is not a matched full-benchmark baseline. Replaying a published edit also differs from independently discovering it. The card supplies useful reproduction evidence without closing either gap.