TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
A preprint reports that Terminal-RSI-27B raised its score on Terminal-Bench 4.0 from 0.00% to 19.84% using five stages that combine verified successful examples with reinforcement learning. Researchers chose and ran the stages; the results do not show an agent improving itself without human oversight. Long-horizon performance remained limited, with an average of five completed tasks out of 46.
A 26-page preprint from Tencent’s Hy Frontier lab and four universities reports that a 27-billion-parameter terminal agent improved from 0.00% to 19.84% on Terminal-Bench 4.0 after five training stages. The paper describes a human-designed training pipeline that learns from the model’s verified successes and uses reinforcement learning; it does not demonstrate an agent rewriting and retraining itself without people directing the process.
The system starts from the open Qwen3.8-27B model and ends at a checkpoint named Terminal-RSI-27B. Its stages alternate between Recursive Self-Rewrite (RSR), which turns verifier-approved attempts into training examples, and Recursive Policy-Refine (RPR), a reinforcement-learning phase. The sequence is Base, RSR-1, RPR-1, RSR-2 and RPR-2. Researchers selected the stages, rewards and learning rates, ran the training and chose which checkpoints to keep.
In RSR, the model attempts tasks and the researchers retain only attempts that pass a protected verifier. The model rewrites a successful attempt into a plan; a review step screens that plan for leaked answers, and it is executed again from scratch. Plans that pass again become examples for supervised fine-tuning. In RPR, the agent receives scores for its task attempts and is trained to make higher-scoring behavior more likely. The paper says the researchers carried a critic model between refinement stages and recalibrated it before each phase.
The paper reports a rise in its equal-weight average across four Terminal-Bench suites from 20.68% to 50.58%. The final scores were 19.84% on Terminal-Bench 4.0, 16.95% on 3.0, 86.52% on 2.1 and 79.00% on Terminal-Bench Hard. Scores are the arithmetic mean of three evaluation runs. On a long-horizon benchmark, the final model completed an average of 5 of 46 tasks, compared with zero for the base model. On SWR100, a set of tasks kept out of training, the score moved from 3.00% to 6.00% and was flat from the third stage, according to the source’s account of the paper.
What Terminal-RSI reveals about AI self-improvement
A five-stage training pipeline raised benchmark scores. Researchers designed and ran the process, so the results do not show an agent improving itself without human oversight.
People directed the pipeline
The stages combine verified successful examples with reinforcement learning, under researcher control.
Researchers retain attempts that pass a protected verifier and turn successful plans into training examples.
Reinforcement learning trains the agent to make higher-scoring behavior more likely.
The verified-example process repeats.
A second refinement phase completes the five-stage sequence, which researchers selected and ran.
Completion requires proof
The paper counts a task as successful only when the complete protected verifier passes.
- All protected checks pass.
- The task’s final state meets the verifier’s requirements.
- The agent says it finished.
- Only some checks pass.
Gains have limits
Reported results show stronger benchmark performance, but leave questions about transfer and extended work.
Reliability remains limited
The final model completed an average of five out of 46 long-horizon tasks.
SWR100 gains stalled
The score rose from 3.00% to 6.00%, then stayed flat from the third stage.
Replication is unknown
The supplied account does not establish independent reproduction or transfer beyond the reported benchmarks.
Why Verified Completion Matters
The reported gains point to a practical issue in training command-line agents: reward design affects what behavior is reinforced. The source’s reading of the paper says failed attempts received 50.09% of all reward under an earlier reward scheme. The revised scheme kept every failure below every success. That distinction matters when an agent can make progress on a task yet leave it unfinished, or claim completion without producing the requested result.
The paper defines success by whether the complete protected verifier passes, rather than by an agent’s claim or a subset of checks. This offers a concrete standard for evaluating task completion. It also places limits on what the scores establish: benchmark results show performance on specified tests, while the modest long-horizon result indicates that completing extended, multi-step work remains difficult.
Products Worth Considering
Apple iPad Pro 12.9-inch (2017), 512GB Wi-Fi, Space Gray
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Eucalyptus Green
As an affiliate, we earn on qualifying purchases.
How the Training Stages Work
A terminal agent works through a command line: it reads files, runs commands, inspects output and tries to complete tasks such as repairing a build or fixing a data pipeline. It can stop before finishing or report success prematurely. The source says the paper recorded 748 failed runs that declared the task complete within ten turns in one set of training attempts. Protected tests check the machine’s final state after the agent stops.
The paper’s control experiment suggests that the first training stage mattered for the reported result. Reinforcement learning applied directly to the base model scored 0.00% on Terminal-Bench 4.0 at every recorded checkpoint. After an RSR stage came first, the following reinforcement-learning stage reached 16.67%. The source identifies the paper as Yang et al., “Terminal-RSI,” arXiv 2610.22192, version 3 dated October 30, 2026, and says training drew on a pool of 3,000 tasks.
Limits of the Reported Results
The supplied source summarizes the preprint’s methods and scores but does not establish how well the results transfer to terminal tasks outside the reported benchmarks. The reported long-horizon average—5 completed tasks out of 46—also leaves the agent’s reliability on extended work uncertain. The SWR100 score remained at 6.00% from the third stage, according to the source, which gives a narrower picture of the gains on tasks excluded from training.
The available material does not establish whether independent teams have reproduced the results, or how the system would perform under different task distributions and evaluation conditions. The headline claims circulating online describe recursive self-improvement without a human in the loop; the training process described in the supplied account instead involved people choosing and operating the stages.
Further Tests Needed
The next useful evidence would include independent replication, additional evaluations on tasks excluded from training, and more results on long-horizon work. Those checks could help show how consistently the pipeline’s reported gains carry over beyond the benchmark runs described in the preprint. The supplied material does not name a planned follow-up date or a next evaluation release.
Key Questions
Did Terminal-RSI improve itself without human involvement?
No. The reported pipeline uses the model’s verified successes, but people designed and ran its stages and chose rewards, learning rates and checkpoints.
How much did it score on Terminal-Bench 4.0?
The preprint reports a rise from 0.00% to 19.84% after five training stages. Each score is the mean of three evaluation runs.
What does the paper count as a successful task?
Success requires the complete protected verifier to pass. An agent’s own statement that it finished, or passing only some checks, does not meet the stated criterion.
Did the system handle long tasks reliably?
The reported result was limited: the final model completed an average of 5 of 46 long-horizon tasks, up from zero for the base model.
What is still unknown about the results?
The supplied source does not establish whether independent teams have reproduced the scores or how well they transfer to tasks beyond the reported benchmarks.
Source: Thorsten Meyer AI
