TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
Z.ai’s GLM-5.3-FlashX-Legal recorded a 19.17% all-criteria pass rate on Harvey’s Legal Agent Benchmark, according to Vals AI’s leaderboard, where it ranked seventh of 81 models. The score measures tasks completed without missing any rubric criterion; the same run passed 92.5% of individual criteria. Z.ai calls the result state of the art among open-source models, but the source material does not establish that comparison set or show published weights for the legal model.
Z.ai, formerly known as Zhipu, describes GLM-5.3-FlashX-Legal as a model for professional legal workflows, including document understanding, legal reasoning and task execution. Its developer documentation says the model is intended for research, contract review and drafting, including initial drafts. Z.ai says it was built on GLM-5.3-Flash and underwent legal-domain mid-training and post-training, with reinforcement learning across tens of thousands of task environments modeled on legal work. The documentation page is undated; Z.ai’s AutoClaw blog announcement is dated October 13, 2026, while Vals AI lists October 12 as the release date.
The benchmark result uses all-pass grading: a task counts as complete only when every criterion in its rubric is met. Vals AI’s archived leaderboard lists the legal model at 19.17% and reports a test cost of $0.15. Z.ai rounds the result to 19.2% and says it improved by 12.5 percentage points over the base model. The same-table comparison supports that improvement, according to the source material. Z.ai also calls the result state of the art among open-source models, but does not name the comparison set; the source material reports no published weights for the legal model.
Z.ai recommends using the model as an agent connected to MCP services for laws, cases and company data. Its pricing page lists $0.37 per million input tokens and $1.25 per million output tokens. The benchmark’s $0.15 test cost is a separate figure from those per-token rates.
Legal AI · Benchmark Brief · 2026
Z.ai’s Legal AI Benchmark Results: About 1 In 5 Tasks, Near The Top
GLM-5.3-FlashX-Legal ranked seventh on Vals AI’s leaderboard. Its task score measures complete rubric passes, while a separate criterion score shows how often individual checks passed.
Strong on individual checks; stricter at the task level
A task counts as complete only when every required item in its rubric passes. One missed citation, deadline, or recommendation can fail the entire task.
How to read it: 92.5% does not mean the model passed 92.5% of tasks. It refers to individual rubric criteria; 19.17% is the share of tasks where every criterion passed.
A structured test of legal work products
The benchmark measures performance on defined assignments under a specific rubric and evaluation setup—not general legal usefulness or suitability for a particular firm.
1,250 tasks
Harvey’s public benchmark spans 24 legal practice areas. Each task pairs an instruction and client matter with a requested work product.
75,000+ checks
Expert-written binary criteria cover facts, conclusions, citations, severity, recommendations, deadlines, dollar amounts, and formatting.
120 held-out tasks
Vals AI says its private holdout mirrors the public task mix, disables internet access, and uses two language-model judges. The final score averages their task pass rates.
Harvey released the benchmark as open source on May 6, 2026. Its grading protocol gives no partial credit at task level: every criterion must pass.
What Z.ai says—and what the cited material shows
Z.ai presents the system for professional legal workflows. Some broader claims remain difficult to assess from the available details.
“On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%.”
— Z.ai developer documentationBuilt for legal workflows
Z.ai says GLM-5.3-FlashX-Legal builds on GLM-5.3-Flash, with legal-domain mid-training and post-training, plus reinforcement learning across tens of thousands of legal task environments. Intended uses include research, contract review, and drafting.
| Question | Available evidence | What remains unclear |
|---|---|---|
| Open-source state of the art? | Claim made by Z.ai | Comparison set is not named |
| Published model weights? | None reported in source material | Open-source status cannot be assessed from these details alone |
| Current score and version? | Vals AI lists the benchmark score; announcement dated Oct. 13, 2026 | Documentation page is undated; no later update is confirmed |
From benchmark task to a scored result
The published score comes from a defined sequence: submit work, grade against a full rubric, then aggregate task-level outcomes.
Benchmark performance is not firm readiness
The leaderboard does not establish how the model will perform on a specific firm’s matters, or whether its data handling meets the firm’s obligations. Z.ai describes outputs as drafts for legal professionals. The reviewed pages did not include a “not legal advice” notice.
Questions legal teams should resolve
Evaluate the model on the work, sources, and controls that matter to your own practice.
- Test on representative firm documents and workflows.
- Verify every citation, authority, date, and quoted holding.
- Set review standards for drafts and recommendations.
- Clarify client-data processing, applicable rules, and access controls.
- Review how MCP connections access laws, cases, and company data.
- Distinguish benchmark run cost ($0.15) from token pricing.
Z.ai’s pricing page lists $0.37 per million input tokens and $1.25 per million output tokens. These rates are separate from the reported benchmark test cost.
Useful signals, with boundaries
The source material supports a leaderboard result and a small independent spot check. Neither settles broad reliability claims.
Four tasks reviewed
A separate four-task check described by Thorsten Meyer AI reported citations to rulings that could not be found in the legal model’s outputs. This small, non-benchmark sample cannot establish a general error rate.
Not a controlled study
In that check, OpenAI and Claude models did not invent rulings; two Claude models cited a real ruling for the opposite of its holding. These observations apply only to the cited tasks.
No next update dated
The source material does not identify a further benchmark milestone or scheduled date for new results. Future documentation may add evidence, but no update is confirmed here.
What the 19.17% result does—and does not—say
Did the model pass 92.5% of tasks?
No. That figure applies to individual criteria. The all-criteria task pass rate was 19.17%.
How did it rank?
Seventh among 81 models on Vals AI’s archived leaderboard. The listed benchmark run cost was $0.15.
Is it confirmed as the best open-source legal model?
Z.ai makes a state-of-the-art claim, but the comparison set is not specified and no published weights are reported in the source material.
What should teams verify first?
Assess performance on their own work, verify citations, and establish data handling, applicable rules, and access controls.
The score is a measure of complete rubric performance on a defined benchmark. It is not a general estimate of legal accuracy or readiness for unsupervised use.
A High Bar for Legal Task Completion
The score highlights a gap between criterion-level accuracy and completing a full legal work product to a strict rubric. A model can pass most individual checks yet fail a task because a single required element—such as a citation, deadline or recommendation—is missing or incorrect. The benchmark’s rules make its 19.17% task score a measure of fully completed assignments, not a general estimate of how often the model produces useful legal text.
For law firms considering the system, the result points to the need to verify output and citations before relying on drafts. The source material also raises procurement questions about how client data is processed and under which rules. Z.ai describes outputs as drafts for legal professionals; the reviewed Z.ai pages did not include a “not legal advice” notice. The benchmark does not, by itself, establish how the model will perform on a particular firm’s matters or whether it meets that firm’s data-handling requirements.
Products Worth Considering
Jackery HomePower 3600 Plus Portable Power Station, 3584Wh, 3600W
Bose QuietComfort Headphones (2nd Gen), Wireless Over-Ear Noise-Cancelling Headphones, Hazelnut Taupe
Logitech MX Brio Ultra HD 4K Webcam, Graphite
As an affiliate, we earn on qualifying purchases.
How Harvey Scores Legal Agents
Harvey released its Legal Agent Benchmark as open source on May 6, 2026. The public benchmark contains 1,250 tasks across 24 legal practice areas. Each task combines an instruction, a client matter containing relevant materials and a requested work product. Harvey says expert-written rubrics contain more than 75,000 binary criteria covering items such as facts, conclusions, citations, severity ratings, recommendations, deadlines, dollar amounts and formatting.
Harvey’s grading protocol gives no partial credit at the task level: every criterion must pass for the task to count as complete. Vals AI says its public leaderboard uses a private holdout set of 120 tasks that mirrors the public mix, with internet access disabled. Two language-model judges grade each submission, and the final score averages their task pass rates. These design choices define what the leaderboard measures; results may not capture every way legal professionals assess usefulness or risk in practice.
““On Harvey LAB, a benchmark for legal agents, GLM’s legal capabilities achieve an all-criteria pass rate of 19.2%.””
— Z.ai developer documentation
Questions Beyond the Leaderboard Score
The source material does not identify the comparison set behind Z.ai’s open-source state-of-the-art claim, and it reports no published weights for GLM-5.3-FlashX-Legal. The benchmark result also does not establish performance across all legal work, firms or client populations. The documentation page carrying the benchmark statement has no date, and the available material does not say whether the model or benchmark score has changed since the announcement.
A separate four-task spot check described by Thorsten Meyer AI reported citations to rulings that could not be found in the legal model’s outputs. That was a small, non-benchmark check and cannot establish a general error rate. In the same check, OpenAI and Claude models did not invent rulings, while two Claude models cited a real ruling for the opposite of what it held. Those observations are limited to the cited tasks and are not a controlled comparison.
What Firms Should Verify Before Use
The immediate next step for firms evaluating GLM-5.3-FlashX-Legal is to examine its performance on their own documents and workflows, including whether citations can be verified and whether drafts meet internal review standards. They will also need to clarify client-data handling, applicable rules and access controls before connecting the model to legal or company information, as Z.ai recommends.
The source material does not identify a further benchmark milestone or a date for new results. The leaderboard, product documentation and any future evaluation details may provide additional evidence, but no scheduled update is confirmed here.
Key Questions
What does GLM-5.3-FlashX-Legal’s 19.17% score mean?
It means the model passed every required rubric criterion on 19.17% of benchmark tasks in the Vals AI leaderboard run. A task failed if even one criterion did not pass.
How did it rank on the leaderboard?
Vals AI’s archived leaderboard placed the model seventh among 81 models. The source material reports a $0.15 cost for the test run.
Did the model pass 92.5% of tasks?
No. The reported 92.5% applies to individual criteria, while 19.17% is the share of tasks on which every criterion passed.
Is it confirmed as the best open-source legal model?
Z.ai claims state-of-the-art performance among open-source models in this evaluation. The source material does not specify the comparison set, and it reports no published weights for the legal model, so that claim cannot be assessed from the available details alone.
What should legal teams check before using it?
Teams should review outputs and citations, assess performance on their own workflows, and establish how client data is handled and which rules apply. Z.ai describes the output as drafts for legal professionals.
Source: Thorsten Meyer AI
