🔍 Read the full analysis: 3 AI Tools, 1 Home Test: Weight-Space Editing With Zero Legal Risk on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
A home experiment used directional ablation to reduce looping in the local model Ornith 1.5 35B-A3B. On 10 published Lexassay tasks, the edited model answered all 10, up from 4 on the same experimental setup, but its legal score remained zero and its answers included 55 wrong or invented citations. The result suggests the edit improved completion, not legal knowledge, and has substantial methodological limits.
A home experiment reported that editing the weights of a local AI model helped it finish more legal benchmark tasks but did not improve its legal accuracy: Ornith 1.5 35B-A3B answered 10 of 10 published tasks after the edit, compared with 4 of 10 on the same experimental setup, while its legal score remained zero. The test, described by Thorsten Meyer on November 5 and updated November 9, used directional ablation, a known technique, rather than the AI-driven weight editing reported by MIT researchers.
The experiment targeted a specific failure: Ornith repeatedly generated near-duplicate reasoning and failed to finish two tasks before reaching a 62,000-token limit. Researchers measured an internal direction associated with the loops, then removed it at half strength from the output weights of 27 of the model’s 40 layers. The work involved no additional training, new legal data, or a model directly reading another model’s weights as text.
On the 10 published Lexassay tasks, the edited model completed all 10, compared with 4 of 10 in the same experimental setup. But graders found only 7 of 50 checked criteria met, up from 3, and counted 55 wrong or invented citations in the edited answers. The reported legal score therefore stayed at zero. The benchmark checks legal references against primary sources and tests questions on German, US and EU law.
The author also reported that an unchanged model using sampling settings recommended by its model card answered 9 of 10 tasks. That result indicates the outcome depends partly on the run settings, not only on the weight edit. A separate published run of the unchanged model answered 8 of 10, but it used a different setup and is not directly comparable with the edited run.
Infographic 01 · What we changed
One loop direction, weakened at half strength
Inspired by the MIT post, but a simpler, classic technique (directional ablation): code written by a Claude Opus 5.5 session measured one loop direction by running the model and projected it out of the weights. No training, and no AI reading weights as text.
The six steps
- Write test material that is not the benchmark: 24 legal development questions plus 8 general controls, 0 overlap with benchmark prompts
- Find the true loop onset at token level: from there, at least 90% of the next 1,500 positions repeat earlier 32-token sequences
- Run the model and measure one loop direction per layer: average internal signal inside the loop minus the reasoning just before it
- Weaken it in the weights:
W ← W − α·r·(rᵀW)on the MLP output matrices of all 256 experts and the shared expert - Bake an edited copy; the original files stay untouched and the edited weights stay internal
- Benchmark original and edit on the same local setup, graded blind by three Claude agents; figures cover the 10 published tasks
Quick test: share of pure copying
Restart 20 tokens before a known loop onset, 8 cases, 3,000 tokens. Lower is better.
The cost of the edit
- Full strength: 23 of 24 legal questions finished, but control answers broke: tokens dropped out of short formulas, and month counting went wrong
- Half strength: 23 of 24 finished, 9 of 9 checked control facts right, and one German control question was answered in Chinese once
- Before the edit: 8 of 24 legal development questions did not finish; all 8 controls did
One run per setting. The loop detector was revised three times during the experiment, so the development numbers come from a moving yardstick. The direction is a statistical description, not a single “loop neuron”.
Sources: our experiment records, 4–5 Nov 2026 (executing Claude Opus 5.5 session; review by GPT-6 Astra in Codex); Ornith-1.5-35B-A3B model card.
Powered by Thorsten Meyer AI
More Completed Answers, Same Legal Score
The experiment highlights a practical distinction for people evaluating AI tools: finishing a response is not the same as answering correctly. A model that no longer gets stuck may appear more useful, but in a high-stakes field such as law, fabricated citations and weak analysis can make a complete-looking answer unreliable.
The result also offers a limited test of a broader idea: if a model’s failure comes from a mechanical obstacle such as looping, editing its internal behavior might help it express capabilities it already has. In this case, the improved completion rate did not produce a higher legal score. The home test therefore supports no claim that weight editing adds legal knowledge or makes a model fit for legal work.
For readers considering local models, the reported findings are a reminder to separate task completion, benchmark scoring and citation accuracy. The experiment involved one model, one small set of published tasks and limited runs. Its result is a case study, not evidence of broad performance gains across models or legal applications.
Products Worth Considering
Samsung Galaxy Tab A11+ 8GB RAM, 256GB Storage, Gray
LG UltraGear 27-inch QHD OLED Gaming Monitor, 240Hz
Amazon Kindle Scribe (3rd Generation), 64GB, 11-inch Display, Graphite, with Premium Pen
As an affiliate, we earn on qualifying purchases.
From MIT Weight Editing to a Home Test
The home test was prompted by an MIT CSAIL post by Rikiya Takehi, Jayden Teoh, Alex Zhang and Omar Khattab. Their work described “weight-space programming,” in which an AI reasons about another network’s weights and edits them directly. In a reported Qwen example, the MIT team said its intervention stopped looping and raised a maths score from 7.9% to 48.1%.
The home experiment did not reproduce that method. Instead, a Claude Opus 5.5 session wrote code to run Ornith, record its internal signals and remove a measured direction from its weights. Meyer described this as directional ablation, a simpler, established approach. He also disclosed that Claude carried out the edit and Claude agents performed blind grading. OpenAI’s GPT-6 Astra, operating in Codex, reviewed the code, figures and claims and identified faults; it was an independent reviewer, not a second experiment.
Lexassay contains 15 legal questions, according to the article: 10 published tasks and five hidden tasks intended to prevent tuning against the full set. The reported figures here concern only the published 10. The article says Ornith, designed for coding and agent work rather than law, scored 1 out of 100 in the broader benchmark. Other local models were not selected: two did not show the same looping pattern, while DeepSeek V4 Flash could not be tested under comparable machine and token-budget conditions.
“The model finishes is not the answer is right.”
— Thorsten Meyer, author of the home experiment
Limits of the Ten-Task Comparison
The reported test has several constraints. There was one run per setting, the development set was small, and the loop detector changed during the experiment. Three graders came from the same model family, though GPT-6 Astra independently reviewed the work and reportedly found errors. These factors limit how confidently the result can be reproduced or generalized.
The unchanged model’s results also varied: it answered 8 of 10 in one published run, 4 of 10 on the setup used for the direct comparison, and 9 of 10 with model-card sampling settings and no edit. Because the setups differed, the figures do not isolate the weight edit as the sole cause of the change. The source says the runs are public, but the five hidden Lexassay tasks were not part of the reported results.
It remains unclear whether the edit would produce similar completion changes in repeated trials, other models, or tasks outside this benchmark. The experiment provides no evidence that the model’s legal answers are reliable; on the reported scoring, its legal score remained zero.
Public Runs and Further Testing
The immediate next step for readers and researchers is to inspect the public run records on Lexassay, including the answers, grades and citation checks, and compare the edited model with the control runs while accounting for their different settings. The article also points to a method page describing how the experiment was carried out.
Further testing would need repeated runs under matched settings and a stable evaluation method to determine whether the edit reliably reduces looping. Any broader claim about legal capability would require stronger performance on validated legal criteria and accurate citations, including evaluation beyond the 10 published tasks. No such result is reported yet; the experiment is described as a single home test, not legal advice or a validated legal tool.
Key Questions
What did the weight edit change?
The experiment removed a measured internal direction at half strength from the output weights of 27 of Ornith 1.5 35B-A3B’s 40 layers, aiming to reduce repetitive looping.
Did the edited model become better at legal questions?
Not according to the reported legal score. It answered more tasks on the direct comparison setup, but its score remained zero; graders counted 55 wrong or invented citations among the edited answers.
Was this the same method used by the MIT researchers?
No. The home test used code to measure model behavior and apply directional ablation. The MIT work described an AI reasoning directly about another model’s weights and editing them.
How strong is the evidence?
The article reports one run per setting on 10 published tasks, with changing loop detection and differing setups for some comparisons. The findings are preliminary and do not establish that the edit will work reliably on other runs or models.
Source: ThorstenMeyerAI.com
