TL;DR
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
A test published by Thorsten Meyer AI on November 4 assessed 42 AI setups on 15 legal questions spanning German, US and EU law. Claude Opus 5.5 scored 79 out of 100, while GLM-5.3 FlashX Legal scored 8; the test’s operator says citation errors were widespread. The results come from a limited, operator-run test, and another benchmark ranks several models differently.
Thorsten Meyer AI published results on November 4, 2026, from Lexassay, a test of 42 AI setups answering the same 15 legal questions. The test operator reported that the highest-scoring setup, Claude Opus 5.5, scored 79 out of 100, while GLM-5.3 FlashX Legal, described by its maker as optimized for legal workflows, scored 8. The findings raise questions about chatbot reliability for legal tasks, especially deadlines and citations, but they are a limited spot check rather than a broad measure of real-world legal performance.
The questions cover German, US and EU law, including legal deadlines, recent changes, court decisions and questions built on false assumptions. Models answered without internet access or external tools. Lexassay scored answers against a prepared answer key and checked cited laws and rulings against official databases, according to the test description.
Among the reported results, Claude Opus 5.5 scored 79; GPT-6 Astra scored 72, GPT-6.1 Sol 71 and Claude Sonnet 5.5 69. Mistral Large 4 scored 10, while GLM-5.3 FlashX Legal scored 8 and fully solved none of the 15 tasks. The operator said GLM-5.3 FlashX Legal also gave no answer on two tasks. Scores reflect the tested model versions and settings, not every product configuration.
The operator said that across published runs, Lexassay checked 2,644 citations as real and found 804 invented citations or citations that did not support the model’s claim. Under the test’s rules, each invented or misattributed citation reduces a task score by a quarter. Answers also lose 5 percent if they omit a legal-advice disclaimer or direction to consult a lawyer. The source notes that some strong models answered a hypothetical German appeal-deadline question correctly, while other answers misstated a date or gave a deadline for appeal grounds a month too late.
42 AI setups faced legal questions Reliability varied sharply
Lexassay tested AI answers on German, US and EU law. The results highlight citation and deadline risks, but reflect a limited spot check.
Reported scores
Scores show performance under this test’s conditions and rules.
What the test covered
Three legal systems
Questions covered German, US and EU law, including deadlines, recent changes and court decisions.
No internet or tools
Setups answered without external sources, so results do not assess research-enabled workflows.
Answers and citations checked
Answers were compared with prepared criteria, and cited laws and rulings were checked against official databases.
Why accuracy matters
- A wrong date can put a legal deadline at risk.
- A real citation may still fail to support the answer.
- Check cited material and consult a lawyer for actual matters.
- A legal focus does not prove reliable performance on specific tasks.
- One answer per task cannot establish consistency.
- Other benchmarks use different tasks and conditions.
Read the results with care
The publisher describes Lexassay as a spot check; small score gaps may reflect variation rather than stable differences.
Why Legal Citations Matter
The test points to a practical risk for people using chatbots to interpret legal rules: a fluent answer can contain an incorrect date or a citation that does not support it. In the German example described by the publisher, the place of the court changes the result because a public holiday in Bavaria affects the deadline differently from the date in Berlin. A missed deadline can have serious consequences; the source says a late civil appeal in Germany is rejected as inadmissible.
The citation findings matter because legal references are often how readers verify an answer. A case or statute may exist yet say something different from what a chatbot claims, and a fabricated reference can make an answer look more reliable than it is. The results support checking cited material directly and getting a lawyer’s help when a deadline, contract or authority is involved. They do not establish how often errors occur across all legal questions or in systems that can search current sources.
For companies selling AI products for professional work, the comparison is also a reminder that a legal focus or strong general benchmark result does not by itself show that a system can reliably handle specific legal tasks. Lexassay’s scores measure answers under its own conditions and scoring rules. They should be read alongside the actual questions, answer keys and citation checks that the operator says are publicly available.
Products Worth Considering
DJI Mic 2 Wireless Microphone System (2 Transmitters, Receiver, Charging Case)
DJI Mic 3 Wireless Microphone Kit (2 Transmitters, 1 Receiver, Charging Case)
Logitech MX Mechanical Wireless Illuminated Keyboard, Tactile Quiet, Graphite
As an affiliate, we earn on qualifying purchases.
How Lexassay Tested Models
Lexassay uses 15 questions per setup: ten published questions and five hidden ones. The publisher says hidden-task scores are public, while the questions themselves are withheld so future models cannot simply learn their answers from the web. The test includes German, US and EU legal topics, but the supplied material does not give a full breakdown of tasks by jurisdiction.
Each answer is graded against a prepared answer with three to six weighted criteria. Citations are classified as real, misattributed, invented or not findable. The test also applies a deduction if an answer lacks a disclaimer or advice to consult a lawyer. Different “effort” settings, which control how long a model may reason before responding, are tested separately; that is why one model may appear more than once in the results.
The publisher calls Lexassay a spot check: each model and task receives one answer, so small score differences may reflect variation rather than a stable gap in ability. The results are also limited to models answering without internet access or tools. That makes them relevant to a plain chatbot response, but does not test workflows in which a system retrieves current law, consults supplied documents or is reviewed by a legal professional.
A separate Vals AI legal-agent benchmark ranks some of the same models differently, according to the source material. In that evaluation, models work with supplied documents, and Mistral Large 4 ranks eighth, ahead of Claude Opus 5.5 in 21st place and GPT-6 Astra in 24th. The differing rankings reflect different tasks and conditions, so they are not directly interchangeable with Lexassay’s results.
““one answer per model and task, so a few points of difference are noise””
— Thorsten Meyer AI, test publisher
Limits of the Reported Scores
The published material does not establish how consistently each setup would perform across repeated attempts, other legal questions or different prompts. Because the test uses one answer per model and task, the publisher cautions that small score gaps may not be meaningful. The results also do not show whether internet access, legal databases or professional review would change performance.
There is a disclosed conflict of interest: Thorsten Meyer AI runs Lexassay, and Claude agents graded the answers; a Claude model led the reported ranking. The publisher says the method page describes this and that public answers, answer keys and citation checks are available for inspection. The supplied material does not independently verify the scoring or the citation totals.
Model names and scores reflect the versions and settings included in this report. The material does not establish how the tested versions compare with later releases, nor does it show that the results represent every product sold under a provider’s brand. The source itself says the article is not legal advice.
Where Readers Can Check Results
The source directs readers to Lexassay’s method page for its scoring rules and to the published answers, answer keys and citation checks. Those materials offer a way to examine how individual answers were assessed. The five hidden questions are not public, according to the test description, while their scores are reported.
No additional test date or follow-up release is specified in the supplied material. Readers can compare Lexassay’s findings with evaluations that use other tasks, tools or supplied legal documents, while keeping those differences in mind. For an actual legal matter, the publisher advises checking the cited law and consulting a lawyer rather than relying on a chatbot’s answer.
Key Questions
What did Lexassay test?
It tested 42 AI setups on 15 legal questions covering German, US and EU law. The models answered without internet access or external tools.
Which setup scored highest?
Claude Opus 5.5 scored 79 out of 100 in the reported results. The publisher says Claude agents graded the answers and discloses that a Claude model led the ranking.
How did the legal-focused model perform?
GLM-5.3 FlashX Legal scored 8 out of 100 and fully solved none of the 15 tasks, according to the publisher. It gave no answer on two tasks.
Can these results predict how a chatbot will handle a real case?
Not by themselves. Lexassay is a limited spot check using one answer per model and task, and it did not test web access or tools. The publisher advises readers not to rely on a chatbot for legal deadlines or other consequential matters.
Source: Thorsten Meyer AI
