Ask a frontier model for research ideas and it will write a hundred before lunch. Which of them deserves an experiment is the expensive question, and it grows more expensive as agents learn to run experiments for days at a time. Most proposals do not survive a run: one automated system generated over 5,000 ideas, judged about 1,100 worth testing, and found 21 that beat the state of the art.
The usual answer is taste: an expert reads the idea and judges whether it is worth pursuing. Taste is real, but it is slow, scarce and personal, and the models now asked to stand in for it are unsteady. Told that reviewers had found one of two identical ideas novel, a current model changed its verdict on more than half of the pairs, and the best of fourteen model judges matched expert preferences 73% of the time, against 51% for random picks.
We take another view. Wherever an experiment can settle an idea, choosing it is a forecast: a bet that the test will pay, by some amount, at some cost. A forecast can be written down before the run and scored after it, so taste stops being a matter of opinion and becomes a record that can be checked, compared and learned from (figure 1).
SchematicFigure 1. Two ways to judge an idea
We test that view on two records from ECDSA.fail, a public benchmark where people working with AI agents publish verified improvements to a quantum circuit. The first is the benchmark’s 300 most recent accepted entries, each with a public note that names the model behind it; we read every note in full and coded how its idea was found. The second is our own loop’s trials on the same benchmark, each with odds written before it ran, and a replay of 90 of them through a general-purpose forecasting model.
What a forecast is made of
A forecast for an idea has three parts, written before anything is built. The odds are the chance the test pays at all. The size is how much it pays if it does. The price is what the test costs—minutes of an agent, hours of compute. A good chooser ranks ideas by odds times size over price, not by odds alone, because in research a few large wins carry most of the gain: on this benchmark, 8 of the 300 entries made 61% of the fall.
To score odds we use the Brier score, the average squared gap between the odds and what happened. It is 0 for a chooser that is always right and sure of it, and 0.25 for one that says 50% to everything. A second measure, ranking, is the chance a chooser puts a trial that paid above one that did not: 0.5 is a coin toss and 1 is perfect. A chooser can rank well and still give poor odds, and the difference matters as soon as you must decide how much to spend.
Where 300 improvements came from
ECDSA.fail scores a circuit for one elliptic-curve point addition, the core of Shor’s algorithm against elliptic-curve signatures, by the Toffoli gates it runs times the qubits it needs; lower is better. An entry is promoted only if it beats the best score at the time, so every promoted entry is a measured improvement. The 300 most recent, from 12 June to 11 October 2026, took the best score from 1,703,067,769 to 944,250,852, a fall of 44.6%.
Six model readers took 50 entries each, read every note in full, looked at the code where a note was thin, and coded how the note says the idea was found. We grouped those ways into two kinds. Searching near the best starts from the current design and varies it: sweeping settings, scoring many machine-made candidates, taking an idea from another entry, merging branches. Understanding the design starts from a fact about it: measuring where the cost sits and going for the largest part, deriving a new identity or bound, or counting room the design leaves unused and taking it.
The board’s numbersFigure 2. Every promoted entry, by how its idea was found
- Understood the design
- Searched near the best
- Other or not stated
- Our loop’s entries
- Mean step
Searching near the best is what most entries did: 179 of 300. It is what an agent does well, and it produced a steady stream of small steps; half of them moved the score by less than 0.02%. Ideas that came from understanding the design were rarer, 62 entries, and larger. With our loop’s entries and the two new designs left out, their mean step was 0.157% against 0.064%, 2.4 times the size, and 17% of them were steps of 0.3% or more against 5%. A gap that large would turn up by chance about twice in ten thousand shuffles of the labels.
The mechanism is simple. A search near the best finds the steps that lie near the best, and on a mature design those are small: a setting fitted a little better, a candidate that saves a few gates. The ceiling shows most clearly in two rows. The 31 entries that took an idea from another entry never made a step of 0.3%; the largest was 0.21%. The 46 that scored many machine-made candidates had a median step under 0.001%. A large step needs a fact the current design hides—which part costs the most, or that two operations can share one pass—and that fact has to be measured or derived before any search can find it. Models left to themselves lean the other way: given the same prior work as 11,683 published papers, nine model settings, Claude Sonnet 4.6 and Gemini 3.1 Pro among them, framed 47% to 64% of their ideas as bridges between existing lines of work, against 12% of the papers.
Seen as shares, the split is starker (figure 3). Searching near the best was 60% of the entries and 20% of the fall; understanding the design was 21% of the entries and 37% of the fall. Most of the rest sits in entries whose notes do not say how the idea was found, including the largest step of the period. The biggest jumps are the least explained, which is itself a reason to keep a record of how ideas are chosen.
The board’s numbersFigure 3. Where the entries went, and where the fall came from
- Understood the design
- Searched near the best
- Other stated
- Not stated
Three checks guard the result. It holds in both halves of the period, at 2.5 and 2.2 times. It holds with our loop’s 12 entries left out, which matters because our loop was built to measure costs first and eight of its entries sit in the understanding group; leaving them out lowers the gap from 2.9 to 2.4 times. And the coding is the readers’ judgement of what the notes say, so a gap of two times means more than one of 10%.
An idea’s worth moves with the design
A verdict on an idea is a verdict on the idea against the design as it stood. Change the design, or learn that one of its prices was wrong, and the verdict can flip. The largest step of our own six-day run on the benchmark shows this from start to finish (figure 4).
The idea was to reduce a register once after three additions instead of after each one. Priced on paper early in the run, it promised 0.2% to 0.4% but needed about 45 extra wires at a measured 155 gates each, so it did not pay. It was not thrown away. It was parked with the one number that would bring it back: it reopens if three leftover wires can be erased for under 20 gates. In the next round a worker was sent that single question, with a forecast that no such erase existed; forty minutes later it had one that costs nothing. A second worker then found that the 155 gates belonged to another part of the design, and that where the new wires were needed a wire cost half a gate. The first full build saved 6,303 gates, and the entry took 1.449% off the best score, under 13 hours after the question was sent.
Schematic, with the record’s numbersFigure 4. One idea, closed and reopened
- The planner’s forecast
- The result
Two things in that path matter for choosing. The planner’s forecasts were wrong in the useful direction—it expected 450 and 800 gates from the two follow-ups, and they saved 3,007 and 1,841—and that was visible only because the forecasts had been written down. And a closed idea was cheap to reopen because its closure carried a number rather than a judgement. The board shows the same churn at scale: after seven of its eight steps of 1% or more, other entries followed within a day.
Scoring the choosers
If choosing is forecasting, then anyone who chooses—an expert, an agent, a model—can be scored on the same trials. On 10 October we took the 90 trials of our run that had a clear outcome, 44 of which paid, and gave each to a commercial forecasting model built for yes-or-no decisions: once with the trial’s own description, and once with the record of every earlier round. We compared its odds with the simplest use of our record, the share of earlier trials at the same stage that paid.
The run’s numbersFigure 5. Four choosers on the same trials
On the trial’s text alone, the model did little better than saying 50% to everything: a Brier score of 0.243 against 0.250. Shown the record, it ranked trials as well as the record’s own rates did, 0.84 against 0.83, so it could tell a likely trial from an unlikely one. But its odds stayed between 0.17 and 0.62. Where it said 0.55, 21 of 23 trials paid; where it said 0.38, 23 of 66 did. A chooser that will not say how likely a likely trial is spends too little on the trials that matter, and its score shows it: 0.206 against 0.149 for the record.
Two caveats cut in opposite directions. The record’s advantage comes mostly from stage—41 of 50 builds paid, against 3 of 40 trials that only priced an idea on paper or in a small model—and stage is partly an outcome, since a paper trial that finds a saving goes on to a build. That flatters the record. The ledger text the model read was written after each trial, which flatters the model.
Our own planner erred the other way. On the 48 trials of the run with written odds and a known outcome, its odds averaged 0.30 where 46% came true, and its size forecasts ran low as well: results beat forecasts in 19 of 26 trials. Agreement is no comfort either—the planner and its workers erred on the same side in 10 of 13 matched pairs. What a record adds is not a better guesser but a running correction: every forecast is scored, so the next one starts from a measured rate rather than a feeling.
The few studies that test this directly point the same way. A small open model fine-tuned on benchmark outcomes picked the better of two research ideas 77% of the time on held-out pairs from the benchmarks it had learned from, against 61% for GPT-5, and lost the advantage on pairs from other fields, where both scored 46%. Judgement learned from outcomes beats judgement read off text, inside the field it was learned in.
What the odds were worth
If a few large wins carry the gain, a chooser might be tempted to favour long shots. Our record does not support that. Trial by trial, those given odds under 0.35 kept 20 gates per worker-hour against 921 for those given 0.35 to 0.65, and none of the ten largest trials had odds below 0.4 (figure 6).
The run’s numbersFigure 6. What the written odds were worth
The picture changes at the head of a chain. The two pilots that reopened the idea in figure 4 were given 0.30, and the trial they led to was the run’s largest. Counted with the builds they led to, low-odds chains earned 363 gates per worker-hour against 682. The replay says the same: of the 30 trials with the lowest odds by the record’s rates, only 3 paid, and those 3 were the pilots of the run’s two largest wins. So rank by odds times size over price, keep a cheap long shot in every round, and never cut on odds alone.
What did not work
- Hunting for one large step. A push for a third step of 1% mapped 38 concepts from eight other fields onto the design’s costs and reopened closed ideas. Nothing reached 0.5%. The next ordinary round produced the largest step of the run’s second half.
- Designs from a blank sheet. Four designs started from scratch all lost, at 1.49 to 4.04 times the accepted score.
- A general model as a gate. In a live round, the forecasting model flagged 5 of 22 proposed ideas as repeats of closed ones. The scouts who wrote them had already cited all five.
- Search with no new idea. On the board, 43 entries were nothing but a search for a luckier random seed. Together they made under 0.1% of the fall.
The strongest objection
The strongest objection is that this works only where an experiment is cheap and exact, and most of science is not like that. It is largely right. ECDSA.fail checks every entry exactly, and every promoted entry is a measured gain. Where a result takes months, or where a question is worth asking for what it would mean rather than what it would pay, a forecast still exists but settles slowly, and until it settles the judgement stays with people. The claim that survives is narrower: where an affordable experiment can settle an idea, taste can be written as a forecast and scored, and it should be.
- Coded from notes. How each idea was found comes from the solvers’ own notes, read by models. Notes can leave things out, and 52 do not say at all.
- A moving design. Young designs are easier to improve. The gap holds in both halves of the period, but this is an observational comparison, not a controlled one.
- Our own entries. Our loop measures costs first by design. Leaving its entries out lowers the gap from 2.9 to 2.4 times; it does not remove it.
- Small samples. The replay used 90 trials and one forecasting model. The planner’s odds come from 48 trials of one run.
- One benchmark. Every number about choosing comes from one task with an exact public check.
What this means for how we build
Pensieve is built on this view. A person sets the goal and steers; every trial carries a forecast written before it runs and keeps it beside the result. That makes every chooser in the loop—the person, the agents and the models—scorable on the same record, and it turns the record into the material the next chooser learns from.
What would change our mind
- If, on a second benchmark, ideas that come from understanding the design are no larger than ideas found by searching near the best, the board result is a fact about this circuit, not about research.
- If experts’ ratings of ideas before a run predict results as well as forecasts scored on a record, scoring adds little over taste.
- If a general-purpose model given the same record sets odds as well as the record’s own rates, a model trained on outcomes is not needed.
- If, where checks are slow or noisy, forecasts cannot be settled within a working budget, the approach reaches less far than we think.
Figures 2 and 3 plot the benchmark’s public record: 300 promoted entries, their notes and code, read and coded by six model readers on 11 October 2026. Figures 5 and 6 and every number about our run come from the run’s ledger, computed for our paper and for the replay of 10 October 2026. Figures 1 and 4 are schematics; figure 4 carries numbers from the ledger. Other entrants and the forecasting model are not named. The external sources were checked against their full text on 28 October 2026.
Sources
- Ghosh, Recursive Priced Falsification: Asymmetric Trials with a Forecast-Scored Planner for Autonomous Discovery, October 2026. The source for every number about the run. Our preprint; self-reported
- Long, Pender, Huang et al., ECDSA.Fail: Open Autoresearch for Optimizing Elliptic-Curve Point Addition in Shor’s Algorithm, arXiv:2609.09582, September 2026. The benchmark, its score (peak qubits times average executed Toffoli count) and its rules. Preprint, from the benchmark’s organisers
- Brier, Verification of forecasts expressed in terms of probability, Monthly Weather Review 78(1), 1–3, 1950. The Brier score. Peer-reviewed
- Weng, Zhu, Xie et al., DeepScientist: Advancing Frontier-Pushing Scientific Findings Progressively, arXiv:2509.26603, ICLR 2026. Over 5,000 ideas, about 1,100 tested, 21 that beat the baseline (section 4.3). Peer-reviewed
- Sternlicht, Shahid, Jansen, Weld, Siangliulue and Hope, Old Ideas, Novel Problems: The Instability of LLM-Based Novelty Evaluation, arXiv:2610.02022, October 2026. A one-sentence hint changes a judge’s verdict on more than half of identical pairs; retrieval helps little. Preprint
- Chen, Zhao, Fu et al., Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment, arXiv:2608.29696, August 2026. Best of 14 judges at 72.56% soft accuracy on overall quality against expert votes; random 50.57% (table 4). Preprint
- Chen, Zhao and Cohan, Measuring the Gap Between Human and LLM Research Ideas, arXiv:2607.01233, July 2026. Bridge or fragmentation framing in 47.1% to 64.2% of ideas from nine model settings, against 12.1% of 11,683 papers (section 4.2); labels by a model annotator. Preprint
- Mule, Garikaparthi and Patwardhan, Teaching Language Models to Forecast Research Success Through Comparative Idea Evaluation, arXiv:2605.21491, Findings of ACL 2026. A fine-tuned Qwen3-8B at 77.10% against GPT-5 (high reasoning) at 61.10% on 869 in-domain test pairs; 45.67% against 45.96% on 1,410 cross-domain pairs (table 2). Peer-reviewed
Cite this work
@misc{pensieve2026taste,
title = {Taste Is a Forecast},
author = {{Pensieve}},
year = {2026},
month = {October},
howpublished = {\url{https://www.pensievelabs.co/blog/taste-is-a-forecast/}},
note = {Pensieve Labs}
}