🔍 Read the full analysis: We Tried Weight-Space Programming At Home: Claude Edited A Local Model, GPT-6 Astra Checked The Work, And The Legal Score Stayed At Zero on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
An experiment reported by ThorstenMeyerAI.com used Claude-written code to remove a looping-related direction from a local language model’s weights. The model completed 12 of 15 legal benchmark answers after the edit, compared with 4 of 15 in the matched test, but its legal score remained zero and a no-edit settings change produced 13 completions. The small, single-run experiment does not show that the edit improved legal knowledge.
A home experiment reported November 5 by ThorstenMeyerAI.com found that editing the weights of a local AI model helped it finish more legal benchmark answers, but did not improve its legal score. The edited Ornith 1.5 35B-A3B completed 12 of 15 questions, compared with 4 of 15 in the matched unedited run; both scored zero under the benchmark’s scoring measure.
The experiment used code written during a Claude Opus 5.5 session to identify a direction associated with the model’s repetitive output, then subtract half of that direction from the output weights in 27 of the model’s 40 layers. The report says the process involved no additional training, new legal data, or an AI directly reading and editing the weights through reasoning. It describes the approach as directional ablation, a technique distinct from the MIT researchers’ weight-space programming method that prompted the test.
On the same setup, the unedited model completed 4 of 15 answers; the edited model completed 12. But the report says the 12 edited answers included 62 wrong or invented citations, an average of 5.2 per completed answer. Across 75 checked criteria, the edited answers met seven, compared with three before the edit. The benchmark’s overall legal score remained zero in both runs.
A separate settings test complicates the result: with sampling settings recommended by the model card and no weight edit, the original model completed 13 of 15 answers and still scored zero. OpenAI’s GPT-6 Astra, used in Codex, reviewed the code, results and claims and identified faults, the report says. It was an independent review of this experiment, not a second experiment.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, 15 tasks, graded blind by three Claude agents
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
More Completions, Same Legal Score
The result separates a model’s ability to finish a response from its ability to produce an accurate one. For users evaluating local AI systems, fewer loops may make a model easier to use, but a completed legal answer can still contain false citations and fail substantive checks. The reported zero score is a warning against treating fluency or completion rate as evidence of reliability.
The no-edit comparison also means the apparent improvement cannot be credited confidently to the weight change alone. In that test, changing to the model card’s recommended sampling settings yielded one more completed answer than the edited run. The report describes a substantial shift in completions when the local setup changed, from 10 to four for the original model. Those results point to settings and test conditions as important parts of the outcome.
The experiment is therefore a limited demonstration of a possible repair for looping, not evidence that weight edits can make a model legally competent. The article itself says it is not legal advice. Any practical use would need stronger testing of correctness, not just a measure of how often the model reaches an answer.
Products Worth Considering
Bose SoundLink Plus Portable Bluetooth Speaker, Black
Bose QuietComfort Ultra Earbuds (2nd Gen) – Wireless Noise‑Cancelling Earbuds with Mic, Immersive Audio, USB‑C Charging, Up to 6 Hours Batte
EcoFlow DELTA 3 Portable Power Station (1024Wh, 1800W)
As an affiliate, we earn on qualifying purchases.
From MIT’s Weight-Editing Work
The test followed a paper described in the report by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab of MIT CSAIL. Their work calls its approach “weight-space programming”: using a general-purpose model to examine and directly change another network’s weights, without a conventional training loop. In a Qwen example, the researchers reportedly traced looping to a direction in the model’s internal signals and removed it.
The MIT method and this home experiment are related in aim but not identical in how the edit was obtained. In the reported MIT case, GPT-6 Astra reasoned about the weights. At home, Claude wrote code that ran Ornith, measured its internal signals and applied a direction-removal procedure. The report says the test began after the MIT result and ran from Wednesday evening to Thursday morning.
Ornith was selected because it was the local model in the authors’ benchmark that showed the relevant looping problem and fit the available machine. The report identifies it as a 35-billion-parameter mixture-of-experts model, with about three billion parameters active per token according to its model card. The model was built for coding and agent tasks, not law. Its benchmark, Lexassay, asks 15 questions about German, U.S. and EU law without internet access or tools, then checks cited laws and decisions against primary sources.
Limits of the Home Test
The report describes one run per setting, a small development set and a loop detector that changed during the experiment. It also says the graders were three systems from the same model family, while Claude both carried out the edit and supplied the blind grading. GPT-6 Astra’s review provides a check from another company, but does not replace repeated tests or independent human validation.
Because recommended sampling settings alone produced 13 completed answers without an edit, the available results do not establish how much of the completion difference came from the weight change. The report also notes that a change in local setup shifted the original model’s completion count from 10 to four. It does not provide repeated-run statistics or enough detail here to establish whether the edit’s effect would persist across machines, prompts or model versions.
The benchmark results concern this particular model and test, not legal AI systems generally. The article says the published benchmark site still shows only the original Ornith result; the new runs have not been added there. The accuracy of individual responses and the broader reliability of the grading are also not settled by the reported experiment.
Independent Replication Is Needed
The source report does not announce a scheduled follow-up or a planned benchmark update. The immediate next step for readers assessing the claim is to treat it as a preliminary, single-experiment result rather than a validated improvement. Repeating the same comparisons across multiple runs, while holding the local setup and sampling settings constant, would help separate an edit’s effect from ordinary variation.
Further testing would also need to check answer quality and citations, not just whether a response completes. The authors say the new runs are not currently reflected on Lexassay’s public site, which continues to display the original Ornith result. Until additional results are published, it remains unclear whether this weight edit offers a reliable benefit beyond settings adjustments.
Key Questions
What did the experiment change?
Code written during a Claude Opus 5.5 session measured a direction linked to looping and removed half of it from the output weights in 27 of Ornith’s 40 layers. The report says the test did not involve additional training or new legal data.
Did the edited model become better at law?
Not according to the reported benchmark score. It completed more answers in the matched run, but the legal score stayed at zero. The completed answers also contained 62 wrong or invented citations, according to the report.
Could settings have made the difference instead?
Possibly. The unedited model completed 13 of 15 questions using the sampling settings recommended by its model card, compared with 12 after the edit. The report’s single runs do not establish which approach is more reliable.
Was this the same method used by MIT?
No. The report says MIT’s weight-space programming example involved GPT-6 Astra reasoning directly about another model’s weights. The home test used code to run the model, measure internal signals and apply a directional ablation edit.
Can the results be treated as legal advice?
No. The report explicitly says the experiment is not legal advice. Its results are limited, and the edited model’s score remained zero on the stated legal benchmark.
Source: ThorstenMeyerAI.com
