AIThis post was created with the assistance of artificial intelligence (AI).

TL;DR

Specialized AI forecasting systems now match or exceed top human performance on traditional forecasting questions, according to the source material. The next test is whether they can make reliable conditional and causal forecasts, with a new benchmark expected in the coming months.

Specialized AI forecasters now match or outperform top human forecasters on traditional prediction questions, according to the source material, but researchers have not established whether that performance carries over to questions about policy effects or cause and effect. The distinction matters because forecasts about what would happen under different choices could inform decisions in areas such as public health and the economy.

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

The reported performance applies to traditional forecasting questions, such as whether a candidate will win an election. The source points to evaluations including ForecastBench and the Metaculus Cup, and says the strongest results have come from specialized systems built with forecasting-specific processes. It names FutureSearch, Mantic and Preseen as companies working in the area.

The source also says general-purpose systems such as ChatGPT and Claude are about six months behind the specialized forecasters. That is an estimate, not a dated benchmark result or a guarantee that they will catch up by a particular point. The material provides no scores, test conditions or independent evaluation details with which to assess that comparison.

Researchers are now examining conditional forecasts: for example, what might happen to the economy if a particular party wins an election. A Forecasting Research Institute test about a year ago found that AI systems performed significantly worse on such questions than on traditional ones, according to the source. A new experiment is being developed to measure current performance; its results are not yet available.

At a glance
reportWhen: Current status; a benchmark on conditio…
The developmentAI forecasters have reached top-human performance on traditional forecasting questions, shifting research attention to harder conditional and causal questions.
AI Forecasting · Briefing

How AI Is Shaping the Future of Forecasting in Science

Specialized AI systems reportedly match or outperform top human forecasters on traditional questions. Whether they can reliably forecast policy effects or causes remains unproven.

What AI Can Forecast Today

Reported strength applies to conventional questions with checkable outcomes, not every kind of prediction.

Reported strength

Traditional questions

Specialized systems reportedly match or exceed top human performance on questions such as election outcomes.

Examples

Forecasting evaluations

The source points to ForecastBench and the Metaculus Cup, but gives no scores or test details.

Specialists

Purpose-built systems

FutureSearch, Mantic and Preseen are named as companies working on specialized forecasting.

Three Different Forecasting Questions

A forecast about what follows an event is not automatically evidence that the event caused it.

Question typeWhat it asksWhat is established
TraditionalWill a candidate win?Reported top-human performance for specialized AI
ConditionalWhat might happen if a party wins?Current AI performance remains unmeasured
CausalDid the win cause an economic change?Reliable AI causal estimates remain unresolved

Why Conditional Forecasts Matter

Decision-makers need to compare possible outcomes under different choices.

Potential uses
  • Compare health outcomes under different vaccination rules.
  • Consider crime rates under different policing policies.
  • Assess possible economic outcomes under different election results.
Important limits
  • Earlier testing found weaker AI performance on conditional questions.
  • Other factors may influence both an event and its later outcome.
  • A forecast alone does not establish cause and effect.

What Researchers Are Watching

A new Forecasting Research Institute test is being developed to measure performance on conditional questions.

1
Measure current performance

The benchmark is intended to test conditional forecasting; results are not yet available.

2
Compare with forecasters

Findings should clarify how systems compare with human forecasters.

3
Test causal claims

Further evaluation is needed to establish whether forecasts can support causal conclusions.

Key pointAI’s reported edge is limited to traditional forecasting; performance on conditional and causal questions remains unsettled.
The takeaway
Treat specialized AI forecasts as promising for conventional questions, but wait for benchmark evidence before relying on them to assess policy effects or cause and effect.

Testing Forecasts Beyond Elections

Traditional forecasts concern outcomes that can usually be checked directly, such as an election result. Conditional forecasts ask what might happen if a specified event occurs, making them more relevant to choices about policy. A decision-maker might want to compare expected health outcomes under different vaccination rules or crime rates under different policing policies.

That relevance does not make a conditional forecast a causal answer. If an election victory and a later economic downturn occur together, other factors could have contributed to both. A forecast about what follows an election win cannot, by itself, establish that the win caused the downturn. Reliable causal estimates could offer decision-makers a useful tool, but the source does not show that AI systems can yet produce them consistently.

The reported gains on conventional questions therefore mark a starting point, not proof that AI can answer every kind of forecasting question. Whether accuracy holds as questions become more complex will shape how much weight researchers and institutions can place on these systems.

Products Worth Considering

Sony WH-1000XM6 Wireless Noise-Cancelling Headphones – Midnight Blue

Sony WH-1000XM6 Wireless Noise-Cancelling Headphones – Midnight Blue

LG 27GS60QC-B Ultragear 27-inch Curved Gaming Monitor (QHD, 180Hz, 1ms, 1000R)

LG 27GS60QC-B Ultragear 27-inch Curved Gaming Monitor (QHD, 180Hz, 1ms, 1000R)

As an affiliate, we earn on qualifying purchases.

From Standard to Conditional Forecasts

Until recently, people seeking forecasts at the level of top human teams had relatively few options, according to the source. Hiring superforecasters could cost thousands of dollars and take weeks. Prediction markets and platforms such as Metaculus offer another route, but the source describes their question sets as limited and notes that market questions require clear rules for resolving outcomes.

AI systems can be asked questions without setting up a public betting market, and forecasting companies have made conditional questions a feature of their products, the source says. That makes it easier to pose questions about a range of possible decisions. It does not, however, show that an answer is accurate: the earlier Forecasting Research Institute test found a gap between AI performance on conventional and conditional questions.

There is a further complication: a conditional association is not necessarily a causal effect. To estimate cause and effect, forecasters may need to account for factors that influence both the condition and the outcome. The source describes one possible approach as combining forecasts across relevant variables, but reports no validation showing that this method reliably identifies causation.

Conditional Accuracy Still Unmeasured

The source does not identify the exact AI models, dates, sample sizes, scores or evaluation protocols behind the claim that specialized forecasters match or beat top humans. It also does not establish how the reported results compare across different forecasting tasks. The estimate that general-purpose models are about six months behind should be treated as a claim in the source, not as a confirmed timeline.

For conditional forecasting, the current performance level remains unknown: the consulted experts would not say whether AI is better or worse than an average well-informed human, though they thought it remained behind top superforecasters. No published result from the new benchmark is included in the source. And even a strong conditional forecast would not automatically establish causation; the reliability of the proposed approaches to causal forecasting is also unresolved.

Benchmark Results to Come

The next reported milestone is a Forecasting Research Institute benchmark intended to measure AI performance on conditional questions. The source says results are expected in a few months, but gives no publication date. Those findings should help clarify whether systems have improved since the institute’s earlier test and how they compare with human forecasters.

Further evaluation will be needed to determine whether AI forecasts can support causal conclusions, rather than simply estimate what tends to happen alongside a particular event. Until results are available, claims about AI forecasting should be kept specific to the traditional questions on which performance has been reported.

Key Questions

What has AI achieved in forecasting?

The source says specialized AI forecasters now match or outperform top humans on traditional questions, including the types used in ForecastBench and the Metaculus Cup. It does not provide scores or evaluation details.

Can AI forecast the effects of a policy?

That remains unsettled. Conditional forecasting asks what may happen if a policy or event occurs, and an earlier Forecasting Research Institute test found that AI systems performed worse on those questions than on traditional ones.

Does a conditional forecast prove cause and effect?

No. A forecast about what may happen after an event does not by itself show the event caused the outcome. Other factors may affect both the event and what follows.

When will researchers have a clearer answer?

The source says the Forecasting Research Institute is working on a benchmark, with results expected in the coming months. It gives no exact release date.

Source: astralcodexten.com

You May Also Like

The Future Of Audio AI: How Grok Voice Realtime And xAI Lead The Way

xAI’s Grok Voice Realtime emerges as a key audio-to-audio model, signaling advances in real-time voice interaction technology, though details remain undisclosed.

Grok API Models Get Invisible Text Watermarking From xAI

Crypto Briefing reports that xAI added invisible text watermarking to Grok API models, but the affected models and technical details are not known.

Altman Shares OpenAI’s Vision Of AI Building Data Centers

A headline says Sam Altman discussed a vision of AI helping build data centers, but offers no details on plans, timelines or implementation.

What Makes The Scroll-Driven Rotor In “TURBINE — WERK 9” Tick?

TURBINE — WERK 9 uses a scroll-driven SVG rotor to connect page movement with an industrial field-guide experience.