📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva LLM, built from scratch with extensive Italian data, underperformed on academic benchmarks despite impressive technical results. This highlights challenges in scaling language models for country-specific knowledge.
Italy’s Minerva-3B, a large-scale European sovereign language model trained entirely from scratch, scored only 4.9% on the INVALSI Italian school exam benchmark, despite extensive data and infrastructure investments. This result challenges assumptions about the relationship between training scale and language understanding, and underscores ongoing debates about sovereign AI development strategies in Europe.
Minerva was developed by Sapienza University of Rome’s NLP group, led by Roberto Navigli, using Italy’s national supercomputing resources and funded through Italy’s PNRR AI strategy. The project trained models ranging from 350 million to 7 billion parameters on 2.5 trillion tokens, with approximately 50% Italian content, making it one of Europe’s most ambitious efforts to create a country-specific LLM from scratch.
Despite this scale, Minerva-3B’s performance on the INVALSI benchmark was strikingly low—just 4.9%, near chance level—indicating that massive data and parameter counts alone may not suffice for complex academic language understanding. Researchers noted that dataset size and parameters are more crucial than pre-training composition for such tasks.
This empirical finding contrasts with the results of the European multilingual approach exemplified by Portugal’s AMÁLIA, which layered specialization onto a multilingual foundation but did not produce comparable language-specific depth, raising questions about the most effective strategies for sovereign AI in Europe.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Large Language Models (LLMs)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
Large-Scale AI Engineering: Design, Train, and Optimize Foundation Models on NVIDIA GPU Clusters
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
KNAT Full Study Guide: Smart Edition Academy Kaplan Nursing Entrance Exam Study Manual with 4 Full Length Practice Tests + 500 Realistic Questions + … + Online videos + Online Flashcards
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
AIOMEST Digital Lux Meter, Illuminance Light Tester Measures 0.1~400,000 Lux/FC, Foot Candles Lumens Detector with Data Storage 270° Rotatable Head for Plants Office LED Luminosity, AI-8801B
☀【0.1~400,000 Lux/FC】Accurately measures light 0.1~400,000 Lux/FC (resolution: 0.1; accuracy: +/- 4% of rdg ), ambient temperature (℃/℉ optional),…
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for European Sovereign AI Strategies
The results from Minerva suggest that simply increasing data scale and model size may not be enough to achieve deep language understanding at the national level. This challenges current assumptions in European AI policy, which often emphasize large-scale training and infrastructure investments. It highlights the need for a nuanced approach that considers the scale of native-language investment necessary to produce models capable of understanding complex, country-specific knowledge and tasks.
For policymakers and AI developers, this means reevaluating the resource commitments and strategies needed to build truly effective sovereign language models. The findings imply that future efforts may require even larger investments or alternative approaches, such as more targeted data curation or hybrid models, to bridge the gap between technical capability and linguistic depth.
European Sovereign LLM Development Approaches Compared
Italy’s Minerva project represents a deliberate choice to train a language model from scratch on a massive, country-specific dataset, contrasting with approaches like Portugal’s AMÁLIA, which focused on continued pre-training of multilingual models with smaller proportions of native data. Minerva’s development involved significant institutional coordination, including Italy’s national supercomputing resources, and was part of broader European efforts to foster sovereign AI capabilities.
Previous efforts in Europe have debated whether to prioritize large-scale, from-scratch training or incremental specialization. Minerva’s results provide empirical data to inform this debate, revealing that scale alone may not guarantee high performance on complex language tasks, especially in academic contexts.
“Minerva’s performance on the INVALSI benchmark highlights the need for a deeper understanding of what scale of native-language investment is truly necessary.”
— Thorsten Meyer
What Aspects of Minerva’s Performance Remain Unclear
It is not yet clear whether further iterations of Minerva, with larger models or different training approaches, could improve performance on complex language tasks. The impact of data quality, targeted fine-tuning, or hybrid methods remains to be tested. Additionally, the broader implications for other European languages and the scalability of this approach are still under investigation.
Next Steps for Minerva and European Sovereign AI
The Minerva team plans to continue refining their models, potentially increasing parameters, exploring different training regimes, and conducting more comprehensive evaluations on complex language benchmarks. Policymakers and researchers will likely reassess investment strategies in light of these findings, emphasizing the need for larger-scale native-language data and possibly new architectural innovations to achieve deeper language understanding.
Key Questions
Why did Minerva score so low on the Italian academic benchmark?
Despite extensive data and large models, the empirical results suggest that scale alone may not be sufficient for complex language understanding, especially in academic contexts. The dataset size and model parameters are more influential than pre-training composition, indicating a need for even larger investments or different strategies.
How does Minerva compare to other European sovereign LLMs?
Minerva trained from scratch on a large native dataset, contrasting with approaches like Portugal’s AMÁLIA, which layered specialization onto multilingual models. While Minerva demonstrates impressive technical capabilities, its low performance on complex benchmarks indicates that scale alone may not guarantee deep language understanding.
What are the policy implications of Minerva’s results?
The findings suggest that European countries may need to commit even larger investments in native-language data and model scaling to develop effective sovereign AI. This could influence future funding, research priorities, and strategic planning in the European AI ecosystem.
Will further development improve Minerva’s performance?
It is currently uncertain. The team plans to continue research, potentially increasing model size and refining training methods, but whether these efforts will significantly boost performance remains to be seen.
Source: ThorstenMeyerAI.com
