Most of what a research loop tries does not work. That is as true of the language-model agents that now improve programs, circuits and their own code as it is of people. How much a failure costs, and how much a success can return, are set by how the loop is built—yet most loops choose what to try by score or novelty, and pay the full cost of an attempt before they learn that it failed.
This post describes the method in our new paper, Recursive Priced Falsification, and a six-day run on a live public benchmark alongside other agent harnesses. The loop made every trial cheap to lose and left what it could win uncapped, so it could afford to lose most of them. It lost 85 of its 133 trials and still produced 70.8% of the benchmark’s progress.
The loop’s forecasts, the odds, size and stake written down for every move before it ran, were an early version of Felicis, the forecasting model we are now building. Felicis is still in training; this run is where its way of choosing moves was first put to a public test.
What current loops leave untested
Published discovery loops share one shape: agents propose a change, an evaluator scores it, and the loop keeps what passes. FunSearch and AlphaEvolve evolve programs this way, and the Darwin Gödel Machine and AIDE² apply it to agents that edit their own code.
Two things are missing from that shape. The first is a price: ideas are ranked by score or novelty, and an expensive one is found out only after it has been built. The second is a test of the planner. What gets scored is always the candidate, never the planner’s picture of the problem, so when a task is badly posed the experiment fails and the idea takes the blame. If the exact check were free there would be nothing to plan; as we argued in an earlier post, it is the expensive part.
The method
The method is an arrangement of agent roles, what each role may read, the tests a claim must pass between them, and a shared record that holds the state of the research (figure 1). No agent’s conversation holds that state, so any agent can be replaced without loss.
SchematicFigure 1. One round, the planner and the shared record
Price before build. A score is usually made of a few resources, and an idea usually spends one to save another. The benchmark we ran on scores a circuit by its qubits times its gates; at 800,000 gates and 1,250 qubits, one qubit is worth 640 gates. An idea that frees a qubit for 500 gates pays, and one that costs 800 does not, however clever it is. The loop measures what each setting of the design trades, keeps those rates in a price map, and prices every idea against it before anything is built. In the run, ten settings sold the same allowance of errors for anywhere from 133 to 994 gates per error.
Two forecasts and a falsifier. Every task is a trial. The planner writes its forecast and the result that would end the idea; the worker writes its own forecast before it starts. The gap between them is information before any compute is spent: if two forecasts differ by a factor of four, at least one is wrong by a factor of two. The falsifier is agreed before anyone has a result to defend, so when it fires the line ends without argument. Four fired in the first fifty hours, and each ended its line within the hour.
Dead ends that keep a number. An idea is closed only on graded evidence, from a price on paper up to a proof; one dropped by argument alone is parked. Each carries a reopening condition written as a number—“reopens if the three leftover wires can be erased for under 20 gates”—which turns a dead end into a question one worker can answer in one sitting. A red team attacks the closed list. Of the eleven closures it attacked, every conclusion held, though two of the planner’s reasons were wrong.
Cheapest refutation first. Tests are ordered by what they cost per idea they kill, not by how strong they are, a classical result. A one-minute model that catches a third of bad ideas goes before a ten-minute build that catches nine in ten: three minutes per kill against eleven. The run’s checks went from integer identities and small models, in minutes, to a search for a passing test, in hours and at $0.19 to $1.77 of rented compute, and finally the benchmark’s judge. Prices on paper and small pilots ended 55 of 64 ideas before anything was built.
Rank by gain per unit of stake. A trial’s stake, the expected cost of its tests, is known before it runs; what it can gain is set by the design. So the loop ranks ideas by odds times size over stake, not by their chance of success: an idea with one-in-ten odds beats a near-certain one if it is thirty times larger at the same stake. Hence the title. Most trials should lose, and they should lose early.
The word recursive names one more rule: the planner is tested like everything else. Every report lists what was wrong in its task. A breaker in a fresh context, shown only the change and the claim, attacks each result before it is kept. After a round that keeps nothing, the planner is rebuilt from the record alone. The loop found its own arrangement of roles the same way: it began as one planner with helpers, gave an outside forecaster no seat once its odds proved less accurate than the record’s, and ran the new arrangement once as a trial with a forecast written first. That round went from an empty list of ideas to an accepted entry in 2 hours 41 minutes.
Six days on a live benchmark
ECDSA.fail asks for the cheapest quantum circuit for 256-bit elliptic-curve point addition, the core of Shor’s algorithm against elliptic-curve signatures. Its organisers call it open autoresearch: people and agents publish verified improvements to a public leaderboard. In practice each entry starts from the best one before it. We ran the loop on Pensieve, our research harness, for 138.7 hours from 7 October, with Claude Opus 5.5 in every role but the breakers, which ran on a second model. Four other entrants had entries accepted in the same hours.
The best score fell by 5.73%. The loop’s 17 entries made 70.8% of that fall, including E4 and E6, the only steps above 1% by any system (figures 2 and 3). It took the top twelve times and held it when the run ended. Eleven other entries used the same model inside general coding agents, the nearest the run has to a baseline without the method; together they made 21.1% of the fall.
The run’s numbersFigure 2. The benchmark over the run
- The loop’s entries (17)
- Other entrants (25)
ScreenshotsFigure 3. The two steps above 1%, on the benchmark’s leaderboard
E4, accepted 8 October at 16:46: 1.04% off the best score
E6, accepted 9 October at 18:27: 1.45% off the best score, and the top of the board
The whole benchmark sped up, from 0.083% a day in the five weeks before to 0.99% a day during the run—twelve times as fast—and then 0.063% a day in the 38 hours after the loop stopped (figure 4). Most of the difference was the loop’s own entries, but the others also moved three and a half times faster than before: they built on its large steps, and answered each of its entries within a median of 3.2 hours. This is not a controlled test. The run began on a design posted seven hours earlier, and young designs are easier to improve.
The run’s numbersFigure 4. How fast the best score fell
- The loop’s entries
- Other entrants
The largest step was an idea the loop had turned down
E6 shows the method end to end. The idea was to reduce a register once after three additions rather than after each one. Priced on paper early in the run, it promised 0.2% to 0.4% against about 45 extra wires at a measured 155 gates each, and an assumed 60 gates to clean up three leftover wires. It did not pay, so it was parked with a number: reopens if those three wires can be erased for under 20 gates.
In the next round the planner sent that one question to a worker, forecasting that no such erase existed. Forty minutes later the worker had one that costs nothing, checked on 4.4 million inputs, and its report questioned the wire price. A second worker found that the 155 gates belonged to another part of the design; where the new wires were needed, a wire cost half a gate per pass. Built one rung at a time, the first full version saved 6,303 gates for 63 minutes of one builder’s time, and two follow-ups saved 3,007 and 1,841 gates where the planner had forecast 450 and 800. The entry was accepted with a step of 1.449%, under thirteen hours after the reopening question was sent.
What the trials looked like
We read every row of the run’s ledger and kept the 133 trials that sought a gain. The median trial cost 50 minutes of one worker and kept nothing, and 85 of the 133 kept nothing at all. Ten trials hold 79% of the points the loop earned (figure 5). Even so, the average trial, losers included, earned 0.03% of the score: the size of the other systems’ median accepted entry.
The run’s numbersFigure 5. Where the points came from
The trials also beat what was written for them: they came true 46% of the time against stated odds that averaged 30%, and returned 1.6 times their forecast size. But long shots did not earn more. Trials given about even odds kept 921 gates per worker-hour against 20 for those given under 0.35 (682 against 363 when each pilot is counted with the build it led to), and none of the ten largest trials had odds below 0.4. What paid was a capped loss against an open gain, not a taste for unlikely ideas. Low odds mattered only at the head of a chain: the pilots that reopened E6 were given 0.30, and a rule that cut on odds alone would have cut them.
What did not work
- Hunting for one large step. A push for a third 1% step reopened closed ideas, mapped 38 concepts from eight other fields onto the cost map and derived the design’s floor again. Nothing reached 0.5%. The next ordinary round produced E12, the largest step of the second half.
- Designs from a blank sheet. Four designs started from scratch all lost, at 1.49 to 4.04 times the accepted score.
- Forecasts that agree. The planner and the workers erred on the same side in 10 of 13 matched pairs. Two forecasts that agree are not thereby right.
- A false claim of exactness. E14 changed no output on 2.17 million random shots, but inputs aimed at its structure separate it from its base: 10,718 shots fail only in the new design and 4,082 only in the old. Whether it raised the failure rate is unknown, and its public note cannot be amended.
What the run does not show
The strongest objection is that one run proves little about a method, and it is largely right.
- No control arm. No run of the same model from the same start went without the method, and the comparison with other entrants is observational: operators, compute and hours differ.
- An easy week. The run began on a design seven hours old, when every system found improvement easier.
- Entries stand on each other. Others built on the loop’s entries and the loop carried changes between designs, so shares of the fall are shares of steps, not of independent work.
- Not recursive self-improvement. The model was frozen. Builds that kept nothing fell from 10 of 35 to 1 of 12 after the full arrangement of roles (p = 0.24), but that is one step at one change, not a trend.
- One benchmark, one harness. The task has an exact public check. The method is untested where the check is slow, noisy or a human reviewer.
All of the research was done by the agents. A person set the goal, approved each submission for the first 86 hours and every rental of compute, and sent occasional short directions, none of them about the circuit. What survives these limits is a narrower claim: where an exact check exists but is expensive, pricing ideas and planning to lose cheaply let one loop do most of the work on a contested benchmark.
What would change our mind
- If the same model in a general coding agent, run from the same start on the same budget, matches the loop’s share, the method is not what did the work.
- If a matched run that ranks ideas by odds alone does as well as one that ranks by gain per unit of stake, the asymmetry is decoration.
- If, where the check is slow or noisy, cheap tests no longer end most ideas, the method reaches less far than we think.
- If later versions of the loop, on matched budgets, do not need fewer trials for each gain they keep, the recursion adds nothing that can be measured. That is the curve our thesis ends on.
Every number about the run comes from the paper, which computes them from the run’s ledger and the benchmark’s public record. Figures 2, 4 and 5 plot those numbers; figure 3 is two screenshots of the benchmark’s leaderboard, with other entrants’ names and pictures blurred; figure 1 is a schematic. Other entrants are not named. The external sources were checked against their text on 16 October 2026.
Read the paper, or run it
The paper gives the method with its proofs, the full evidence and the limits. The method is also released as a skill and a small set of tools that install into a coding agent with one command, npx github:pensievelab/rpf install, and work with Claude Code, Codex, pi or any host that can start a fresh session. The package is not the software of the six-day run, and we have not used it to repeat that run.
Sources
- Ghosh, Recursive Priced Falsification: Asymmetric Trials with a Forecast-Scored Planner for Autonomous Discovery, October 2026. The source for every number about the run. Our preprint; self-reported
- Pensieve, pensievelab/rpf, Apache-2.0. The released skill and tools. Code
- Long, Pender, Huang, Santos, Singh et al., ECDSA.Fail: Open Autoresearch for Optimizing Elliptic-Curve Point Addition in Shor’s Algorithm, arXiv:2609.09582, 9 September 2026. The benchmark and its rules. Preprint, from the benchmark’s organisers
- Romera-Paredes et al., Mathematical discoveries from program search with large language models, Nature 625, 468–475, 2023. FunSearch. Peer-reviewed
- Novikov, Vũ, Eisenberger et al., AlphaEvolve: A coding agent for scientific and algorithmic discovery, arXiv:2506.13131, 16 June 2025. Its evaluation cascade is in section 2. Preprint
- Zhang, Hu, Lu, Lange and Clune, Darwin Gödel Machine: Open-Ended Evolution of Self-Improving Agents, arXiv:2505.22954, 29 May 2025. Preprint
- Srikanth, Zhao, Xu, Wu and Jiang, Recursive self-improvement of AI research agents, arXiv:2609.26457, 22 September 2026. AIDE². Preprint, from the system’s makers
- Simon and Kadane, Optimal problem-solving search: all-or-none solutions, Artificial Intelligence 6(3), 235–247, 1975. The order of tests by cost per refutation. Peer-reviewed
Cite this work
@misc{ghosh2026rpf,
title = {Recursive Priced Falsification: Asymmetric Trials with a
Forecast-Scored Planner for Autonomous Discovery},
author = {Ghosh, Krish},
year = {2026},
month = {October},
howpublished = {\url{https://www.pensievelabs.co/papers/recursive-priced-falsification/}},
note = {Pensieve Labs}
}