Can an AI choose your next experiment before you spend the compute?

Engineering a result is a different test

MLE-bench offers a useful earlier comparison because its tasks require agents to prepare data, train models and run experiments across 75 Kaggle competitions. It uses competition leaderboards to establish human reference points and examines resource scaling and possible training-data contamination. That design measures an extended engineering workflow rather than a single choice among recipes. Its original model scores date from 2024 and cannot rank today's systems.

Our interpretation is that a team should demand both measurements before handing over expensive experiments. A good recommendation can still fail during data preparation, package installation, checkpoint management or final evaluation. Conversely, an agent with modest initial intuition could become useful through careful execution and feedback. Procurement that collapses these stages into one impressive score would miss where staff still need to intervene.[2]

Replication asks whether the experiment can be trusted

PaperBench, released in 2025, asks agents to replicate 20 ICML 2024 papers from scratch. Its author-assisted rubrics divide the work into 8,316 gradable outcomes, and the evaluation includes an automated judge with a separate assessment of judging quality. The resulting replication score concerns implementation and empirical reproduction, rather than predicting a winner. It is another distinct capability test, with its own scope and potential grading errors.

For a research director, the practical question is whether a claimed result leaves enough evidence for someone else to challenge it. We would require the dataset version, split, random seeds, configuration, logs, failed runs and complete evaluation outputs. An attractive graph without that record should not unlock the next project stage. Junior researchers also need time to understand the reasoning, because checking an unfamiliar automated pipeline can become its own substantial task.[3]

Cheap benchmarking depends on what was precomputed

NAS-Bench-101 shows a different route to reducing research cost. Its 2019 dataset covers about 423,000 unique convolutional architectures, repeatedly trained on CIFAR-10, with more than five million trained-model records. Researchers can query those stored results instead of paying for every trial again. The study made comparisons easier to reproduce within a deliberately bounded search space; it did not establish a universal recipe for every task.

The comparison suggests a practical division of labour. Use stored experimental evidence where it matches the problem, ask an assistant to propose alternatives where it does not, and run a small controlled pilot before scaling. A benchmark's convenience becomes a weakness if its task is far removed from a laboratory's data, hardware or objective. A company predicting medical outcomes, for example, should not infer clinical readiness from success on a synthetic training puzzle.[4]

A proposed pilot for real research teams

The newsroom's proposed next test is a prospective comparison on projects selected before any model sees their results. Give an assisted team and an unaided team the same data access, hardware budget and stopping rules. Record the best independently verified result, total compute, researcher hours, abandoned runs and time spent repairing or checking output. Include projects where a simple established method is already strong, so extra complexity must earn its cost. This is an evaluation proposal, not an outcome established by the cited studies.

Teams should also record which recommendations were rejected and why. If a system repeatedly proposes experiments that violate a resource limit, rely on unavailable data or optimise the wrong objective, counting only its successful suggestions would give a misleading picture. Publish results for the complete project denominator and separate assistance from end-to-end autonomy. Repeating the trial in another institution would help show whether the advantage comes from the system or unusually experienced operators.

This matters for access as well as productivity. A smaller university may value avoiding a few wasted runs more than achieving a marginally better leaderboard score. A well-funded laboratory may instead care about repeatable coordination across many projects. Those are different buying decisions and need different evidence. For now, the strongest practical response is to test AI as a documented experiment adviser, then expand its responsibility only where measured work supports that decision. A pilot should document the information boundary too. Give each participant an explicit list of permitted files and tools, record external requests, and separate confidential project data from public benchmark material. Review whether a proposed experiment depends on information that the team is entitled to use. Otherwise an apparent productivity gain could simply reflect unequal access to data, hidden human assistance or a workflow that another institution cannot reproduce under its own constraints.[1][2][3][4]