🔍 Read the full analysis: Perplexity Vs. Intelligence: What Each Means For AI on ThorstenMeyerAI.com
Get business pricing on tech for your team
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
OpenAI has published some internal measurements of its models’ pretraining perplexity and says the metric could help developers compare progress and inform shared safety standards. Its account links lower losses with capabilities observed in its models, but does not show that any specific value guarantees a capability or predicts when future advances will arrive.
OpenAI has published some internal measurements of its models’ pretraining perplexity, arguing that the metric could help AI developers compare progress and contribute to shared safety standards. The figures offer a view of how well models predict text, but do not establish what capabilities a particular score guarantees or provide a standardized comparison across companies.
Perplexity measures next-token prediction on data held out from training; lower values indicate better predictive performance. OpenAI says it has tracked the measure over time, including for models beginning with GPT-3.5, and is now releasing some results it had previously kept internal. It says the decision involved weighing the competitive cost of revealing information about its models against the potential value of industry-wide safety measures.
In its account, OpenAI gives approximate log-perplexity values of 0.99 for GPT-3.5 and 0.91 for GPT-4, and says later models have reached losses below 0.8. The company associates stages in that progression with capabilities including fluent conversation, productive use of chain-of-thought reasoning, and work in mathematics and cybersecurity. These are OpenAI’s descriptions of its own systems, not evidence that a particular score reliably produces any one ability.
OpenAI also says a model at about 0.76 log-perplexity began producing results on open mathematics problems that the post characterizes as groundbreaking. It anticipates further changes in pretraining research as soon as 2027 and new capabilities around 0.68 to 0.7. Those dates, values and capability expectations are forecasts, not confirmed outcomes.
Perplexity and Intelligence A shared measure?
OpenAI published internal pretraining measurements and argues they could help developers compare progress and shape shared safety standards. The numbers offer a signal, not a guarantee of future abilities.
What the metric measures
Perplexity measures how well a model predicts the next token in a held-out dataset; lower values mean better predictions.
A model learns from a large dataset.
Perplexity tracks predictive performance on held-out data.
Prompting, fine-tuning, and reinforcement learning can elicit or shape what a model does.
OpenAI’s reported progression
The company associates lower log-perplexity with capabilities observed in its models; the pattern does not prove a value causes or guarantees an ability.
| Reported level | OpenAI’s account |
|---|---|
| ≈0.99 | GPT-3.5; fluent conversation |
| ≈0.91 | GPT-4; productive chain-of-thought reasoning |
| Below 0.8 | Later models; mathematics and cybersecurity capabilities |
| ≈0.68–0.7 | Range where OpenAI anticipates new capabilities |
Why a shared measure could matter
Task benchmarks can change or saturate as models improve. Comparable pretraining measurements could give researchers and policymakers a common reference.
- Track predictive performance over time
- Compare progress if developers use shared methods
- Inform discussion of common safety standards
- What a model can reliably do
- How safely it behaves
- How much oversight development requires
What remains unresolved
The supplied material does not include the underlying chart or enough detail to independently compare the measurements.
- No standardized reporting or audit process is specified.
- No formal industry standard, timetable, or commitments from other developers are announced.
- The possible 2027 research acceleration and future capabilities are projections, not confirmed outcomes.
If multiple developers publish comparable measurements, researchers and policymakers could gain a consistent way to follow some aspects of model development even as task-specific benchmarks change or become less useful. OpenAI argues that pretraining perplexity could serve as a common reference point for tracking progress and discussing safety standards.
That possibility matters because a shared measurement might make changes easier to identify across time and organizations. But a lower perplexity score is not a full account of a model’s abilities, reliability or risks. It does not, on its own, show how a system behaves in deployment, how much human oversight it needs, or what safety measures are appropriate. Perplexity would be one signal, not a substitute for capability and safety evaluations.
Products Worth Considering

Jackery Explorer 300 Portable Power Station, 292Wh

Bose TV Speaker Compact Soundbar with Bluetooth and HDMI-ARC, Black

Apple Watch Series 12 GPS 42mm Smartwatch
As an affiliate, we earn on qualifying purchases.
From Prediction to Model Capabilities
AI systems are commonly pretrained on large datasets to predict text, then adapted or guided through methods such as prompting, fine-tuning and reinforcement learning. OpenAI’s argument is that pretraining measurements may offer a durable view of progress while downstream tests and benchmarks change as models improve.
The company uses its past models to explain why it tracks the metric, connecting different stages of its reported measurements to abilities observed in systems from GPT-2 through GPT-4 and later models. It also says the appearance of capabilities at particular levels was not predictable in advance. That history may motivate continued measurement, but it does not prove the same relationship will hold for future systems.
OpenAI raises another possibility: AI systems could help with algorithmic research that improves pretraining. The post points to advances in mathematics as a reason rapid progress in deep-learning research may be conceivable, potentially by 2027. It also recognizes that mathematics and deep-learning research are different fields. The proposed acceleration remains speculation, rather than a demonstrated result.
Limits of the Published Figures
The material describing the release does not provide the underlying chart, full dataset details or enough information to independently compare OpenAI’s measurements with results from other developers. It also does not set out a shared reporting method, independent audit process or standard safety decisions tied to particular scores.
It remains unclear how much a given score corresponds to capabilities across different models, training data or development approaches. OpenAI says capabilities can emerge unexpectedly as perplexity falls, but the account does not show that a particular value guarantees a particular ability. The timing and nature of future capabilities—and the possibility of AI-assisted acceleration in pretraining research—are also uncertain.
Whether Other Developers Join
The next meaningful test of OpenAI’s proposal is whether other organizations publish comparable measurements and whether researchers can agree on datasets and methods that make results meaningful across developers. The published account does not announce a formal industry standard, a timetable for creating one or commitments from other companies.
Until those details emerge, the figures are best treated as OpenAI’s internal measurements made public, rather than a basis for ranking developers or predicting capabilities. Further reporting would need to establish how the measurements are calculated, whether independent researchers can review them, and how they relate to practical capability and safety evaluations.
Key Questions
What does perplexity measure in an AI model?
Perplexity measures how well a model predicts the next token in a held-out dataset. Lower values indicate better prediction on that data, but do not by themselves describe every aspect of a model’s abilities or behavior.
What did OpenAI publish?
OpenAI published some of its internal pretraining measurements, including approximate log-perplexity values it associates with models such as GPT-3.5 and GPT-4. The account does not supply enough underlying data to independently compare those results with other developers’ models.
Does a lower perplexity score prove a model has a specific capability?
No. OpenAI links stages in its own measurements to capabilities observed in its models, but its account does not establish that a particular score reliably produces a particular ability.
Is there now a shared AI safety standard based on perplexity?
No. OpenAI argues the metric could contribute to shared safety standards, but the account announces neither a formal standard nor commitments from other developers to report comparable results.
Will new capabilities arrive by 2027?
OpenAI says changes in pretraining research could come as soon as 2027 and forecasts capabilities around certain lower loss values. These are expectations, not confirmed predictions; their timing and accuracy remain unknown.
Primary source: OpenAI · via ThorstenMeyerAI.com
