📊 Full opportunity report: Engineering Is Automated. Research Is the Residual. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
AI systems are increasingly capable of automating core AI engineering tasks, reaching near-saturation in key benchmarks. Research, however, remains less automated, though progress suggests it may also become increasingly mechanized.
Recent developments in AI capabilities demonstrate that AI systems can now automate the majority of AI engineering tasks, reaching near-complete proficiency in several key benchmarks. However, the automation of AI research remains less advanced but is progressing quickly, raising questions about the future division of labor between human researchers and AI systems.
According to Thorsten Meyer’s analysis of Jack Clark’s recent essay, six core benchmarks measuring AI’s ability in research-relevant skills are approaching saturation. For example, the CORE-Bench, which tests research reproduction, has improved from 21.5% in September 2024 to 95.5% in December 2025, with the author of the benchmark declaring it ‘solved.’ Similar progress is seen in the MLE-Bench, where AI performance on Kaggle competitions has increased from 16.9% to 64.4% in about sixteen months, now approaching mid-tier human performance.
Clark’s analysis suggests that the bottleneck in AI research—such as reproducing experiments or competing in ML challenges—is now primarily an engineering problem, not a capability gap. He further notes that advances in kernel design, including automated GPU kernel generation and optimization, are demonstrating that even foundational research tasks are becoming more automated. Conversely, research activities that involve creative hypothesis generation or novel theoretical insights are less mature in automation but are showing signs of rapid progress.
Engineering is automated.
Research is the residual.
Six skill benchmarks. Edison’s framing. The question Clark leaves open is whether research is just engineering at scale.
Jack Clark’s Import AI #455 catalogs six benchmarks measuring AI capability on AI R&D tasks and concludes “AI can today automate vast swatches, perhaps the entirety, of AI engineering.” The residual question is research. The structural read on the residual: it may not be a permanent moat.
Six skills. One trajectory.
Clark catalogs six benchmarks measuring AI capability on AI R&D-relevant tasks. Each individual benchmark could be noise. Six benchmarks moving together is a curve. The pattern is the cascade observed across the broader Clark series — visible here in the specific R&D-skill domain.
AI Tools for Finance and Accounting Professionals: Automate Tasks, Save Hours, Work Smarter
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three data points. Mixed signal.
Clark provides three data points on the creative-spark question. Yes-evidence: Erdős-1051, centaur math discovery, sporadic Move-37-style moments. No-evidence: low yield, framing dependence, absence of acceleration. The mixed signal is the honest read.
The data supports two readings. Pessimistic: rare moments suggest creative insight is qualitatively distinct from engineering work. Optimistic: rare moments are an artifact of low-volume exploration; more shots on goal yields more discoveries. Both readings are consistent with Clark’s “vast swatches, perhaps the entirety” claim. They differ on the residual.
GPU-Accelerated Computing with Python 3 and CUDA: From low-level kernels to real-world applications in scientific computing and machine learning
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five dimensions Clark gestures at but leaves underdeveloped.
Clark’s section is rigorous on the empirical evidence. Five strategic dimensions matter for the institutional response that the Clark series synthesis argues is structurally inadequate.
Ai Automation Kit PLC Programming Software, Logic Function HMI, Run Simulator
1 PLC Controller
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Two readings. Different equilibria.
The structural question Clark leaves open: is research a permanent moat that bounds automated AI R&D, or is it engineering at scale that dissolves with more shots on goal? Both readings are consistent with the current data. They differ by orders of magnitude in consequences.
Productivity multiplier years
Recursive loop operational
Practical MLOps: Operationalizing Machine Learning Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Five audiences. Asymmetric cost of being wrong.
The institutional response should not bet on inspiration being a permanent moat. If the distinction holds, capacity built is still useful. If it closes, capacity is necessary. Asymmetric cost-of-being-wrong points toward building now.
IN INDUSTRY
IN ACADEMIA
POLICYMAKERS
INVESTORS
EVERYONE ELSE
Engineering is automated. The residual is the question. The institutional response should not bet on inspiration being a permanent moat.
Implications of AI’s Growing Engineering Capabilities
The rapid automation of AI engineering tasks signifies a potential shift in how AI research is conducted. As core engineering functions become fully automatable, human researchers may focus more on high-level theory, hypothesis formulation, and creative insight, which remain less automated but are also advancing. This transition could accelerate AI development cycles, reduce costs, and reshape the roles of human researchers in the field.
Recent Benchmarks and AI Capability Trajectories
Jack Clark’s recent essay, analyzed by Thorsten Meyer, highlights six key benchmarks that measure AI’s ability to perform tasks critical to AI research and development. These include research reproduction (CORE-Bench), Kaggle competition performance (MLE-Bench), and kernel design. All six are approaching or have reached saturation points, indicating that AI systems are now capable of performing research and engineering tasks at levels comparable to or exceeding human experts in specific domains. This pattern reflects a broader trend of rapid capability growth over the past 18 months, driven by advances in large language models and specialized AI systems.
While these benchmarks provide concrete evidence of progress, experts caution that the automation of creative and theoretical research remains less certain, though early signs suggest this frontier is also moving forward quickly.
“Clark’s conclusion is correct and possibly understated for engineering. The residual research question is real but may be less binding than the framing suggests.”
— Thorsten Meyer
Uncertainties in Research Automation Progress
While engineering tasks are nearing full automation, the automation of creative research activities, such as hypothesis generation, theoretical development, and novel insight creation, remains less certain. Experts caution that these areas may still require significant human input, though early developments suggest rapid progress is possible. It is also unclear how long it will take before research tasks are fully mechanized or whether new forms of AI-driven research will emerge that differ fundamentally from current human-led approaches.
Next Steps in AI Research and Engineering Automation
Over the next 32 months, focus will likely shift toward refining AI systems capable of automating more complex research tasks, including hypothesis formulation and scientific discovery. Industry and academia may also develop new benchmarks to measure progress in these areas. Additionally, institutional responses may include new policies for AI-driven research, ethical considerations, and adjustments in research funding to accommodate the changing landscape. Monitoring the evolution of these benchmarks and capabilities will be critical to understanding the pace and scope of further automation.
Key Questions
What are the main benchmarks showing AI’s automation progress?
The main benchmarks include CORE-Bench for research reproduction, MLE-Bench for Kaggle competition performance, and various kernel design projects. All are approaching saturation, indicating high levels of automation in engineering tasks.
Will AI fully automate scientific research?
It is uncertain. While engineering tasks are nearing full automation, creative and theoretical research remains less automated but is progressing rapidly. The timeline for full automation is still unclear.
What does this mean for human researchers?
Human researchers may shift focus from routine engineering tasks to high-level theory, hypothesis generation, and creative problem-solving, potentially accelerating AI development cycles.
Are there risks associated with AI automating research?
Potential risks include over-reliance on AI-generated hypotheses, ethical concerns about autonomous research, and the need for oversight to ensure scientific integrity. These issues are currently under discussion.
Source: ThorstenMeyerAI.com
