In March 2024 a start-up called Cognition introduced Devin as “the first AI software engineer”, an agent that could take on a task and finish it by itself. Sixteen months later the same company bought Windsurf, a code editor: the kind of program a person sits in front of all day. Over the same year, the coding tool that grew fastest was one you talk to. Claude Code reached a run rate of a billion dollars a year six months after it became generally available.

Coding agents got very good, and coding still wasn’t handed over to them. What won was a harness: the agent does the work, and a person steers it in conversation, checking as it goes. We think research will go the same way, once agents can reach the places where experiments happen. We call that autonomous, experiment-driven discovery: agents run the experiments by themselves, a person steers, and only an experiment says what has been found. Pensieve is the harness we’re building for it.

Coding was never set loose

The clearest evidence comes from the people who build coding agents. In August 2025 Anthropic surveyed 132 of its own engineers and researchers. They used Claude in about 60% of their work, yet more than half said they could “fully delegate” only between 0 and 20% of it. Asked why, they described working with it step by step and checking what it produced, most of all where mistakes are expensive.

That share is growing. Over six months of Anthropic’s internal usage, the agent made longer runs of tool calls and people took fewer turns. But the shape hasn’t changed: the agent does the typing, and a person stays in the conversation.

Why stay? METR gave an agent 18 real issues from two large open-source projects. Its changes passed the maintainers’ own tests 38% of the time. Then people read 15 of them, and not one could have been merged as it stood. Each needed about 42 minutes of a person’s work, for missing tests, missing documentation or code nobody would want to maintain. That was an early-2025 model, and newer ones do better. The lesson doesn’t age: a check catches what it checks, and somebody has to judge the rest.

The second reason is arithmetic. METR measures an agent by the length of task it can finish half the time, and that length has doubled about every seven months since 2019. Half the time is a coin flip. Chain a few of those stretches together, leave them alone, and the mistakes compound (figure 1).

ModelFigure 1. Leaving a task alone, or checking it as it goes

50%
10

A model, not data. Each stretch comes out right with the same chance, independently. Checked after every stretch, only a stretch that went wrong is done again. Left alone, the whole task runs from start to finish and is run again until a final check passes. Real stretches aren’t independent and real checks miss things, so read the shape, not the numbers.

Ten coin-flip stretches left alone come out right about one time in a thousand. Checked after each one, the same task costs about twenty stretches of work and comes out right every time. Coding settled on conversation for roughly that reason: between stretches, the tests and the person catch what went wrong before it compounds. And in coding that check is cheap, since a test suite runs in seconds. Research has no test suite.

In research, the check is an experiment

In November 2023 Nature published a paper on the A-Lab, an autonomous laboratory at Berkeley. Over 17 days its robots made 41 of 58 target compounds, which the paper called novel. Other chemists soon questioned both how the products had been identified and whether they were new, and in March 2024 they published an analysis concluding that no new materials had been discovered in that work. In January 2026 the journal ran a correction. “Novel” had meant new to the lab’s prediction platform, “not necessarily new to science”, and four of the successes couldn’t be confirmed from the diffraction data alone. The count is now 36 of 57, and one word of the title has changed: “novel materials” became “inorganic materials”.

Running the experiments was the part the machines could do. Saying what had been found took people, and more than two years to settle in print.

That is the difference between research and coding. A research result is checked by an experiment, and experiments take hours or weeks: a training run on rented GPUs, a synthesis in a wet lab, a study with people. They cost real money, and their results often need an expert to read them. Nobody sits beside the agents for every step of that, so a research unit has to work at two speeds. The person steers in conversation when they’re there, the agents run experiments while they’re not, and a record holds where everything stands, so that either side can pick it up. That record is the centre of what we’ve built.

Built for verified experimental progress

Our goal is verified experimental progress: research that moves forward only on results an experiment has confirmed, never on what an agent says it found. Pensieve is the harness for a research unit that works to that rule. You talk to the coding agent you already use, such as Claude Code or Codex, and it drives Pensieve through one command, pensieve. A question with a budget becomes an experiment: pensieve experiment start "<question>" --budget 4h,$40.

Only the check counts. A result is scored by the task’s own check, never by an agent’s account of what it did. In our tests, an agent that rewrote the check still got its real score, so the progress on record is the progress it made.

Every result is traceable. Research methods plug in: the one behind our benchmark run, a general one, or your own. Two can race on the same question, a new one can pick up where another stopped, and every result says which method, in which version, produced it.

Nothing is lost, including what failed. Everything that happens lands in one record, dead ends too, with the result that would reopen them. Leave for a day, ask your agent where things stand, and it reads the same record you do.

Progress you can see. A viewer draws the record as it grows: who did what, how the verified score climbed, and the tables each method keeps (figure 2). Pensieve can also run the agents itself, in the background, several at once and inside the budget. Its first runs on our test problem cost 13 cents.

ScreenshotsFigure 2. Two views of one record in Pensieve’s viewer

The climb: every candidate, scored by the task’s own evaluator, with the lane and the worker it came from

The viewer’s climb: eight candidates scored in order, the best so far drawn as a rising line from 2.54142 to 2.62919, and below it each candidate with its score, its lane (recursive-priced-falsification or general-analysis) and its worker. Opens the full-size image.

The trials: a table the method keeps, one row per worker

The viewer’s table of trials: nine workers over four rounds, scouts and builders, each with its question, its outcome (alive or dead), its result and its gain. Opens the full-size image.
Pensieve’s viewer on a test record we made for this post: two methods side by side on our test problem, packing 26 circles in a square, over three rounds of recursive priced falsification. Stand-in programs play the scouts and builders in place of models, so the trials are scripted, though every score comes from the problem’s real evaluator. Select a screenshot to open it at full size.

The person decides the question, the budget and what counts as better. The agents do the rest, and the record holds both to it. We put an earlier version to the test for six days on a public benchmark that verifies every entry itself.

What one person and a team of agents did in six days

ECDSA.fail asks for the cheapest quantum circuit for the core step of Shor’s algorithm against elliptic-curve signatures, and puts every verified improvement on a public leaderboard. In October we ran a team of agents on it for 138.7 hours, with one method, recursive priced falsification, which prices every idea before anything is built. The best score fell 5.73% over the run. The agents’ 17 entries made 70.8% of that fall, including the only two steps above 1% by anyone, and each counted only after the benchmark’s own verifier had checked it. That is verified experimental progress, measured by someone else.

The person’s part was small and specific. They set the goal, approved each submission for the first 86 hours and every rental of compute (from $0.19 to $1.77 a search), and sent occasional short directions, none of them about the circuit. All of the research was the agents’.

The largest step shows why the record matters. Early on, the agents priced an idea, found it didn’t pay and parked it with a number: reopen it if three leftover wires can be erased for under 20 gates. In a later round one worker was sent that single question. Forty minutes later it had an erase that costs nothing, and within thirteen hours the parked idea was the run’s largest step, 1.449% off the best score. It survived because the record kept the number that would bring it back.

The run had no control arm, it began on a design only seven hours old, and it ran on an earlier version of the harness; the method now ships inside Pensieve. The post and the paper have the full account. And every rental of compute in that run waited for a person’s approval, one search at a time. At scale, a person should set a budget, not sign off on every machine.

The layer between agents and experiments

An agent can design an experiment in seconds. Running it is another matter. A training run needs GPUs that someone has rented and set up, a synthesis needs a lab that will take the order, and a question about people needs people to ask. Each comes with its own interface, its own bill and its own way of reporting a result, and today a person joins them up by hand. Pensieve is the layer that joins them up instead. An agent says what it wants tested and what it may spend; Pensieve runs the experiment, keeps the agent away from the check, and counts the result only when the check holds. That layer is what autonomous, experiment-driven discovery needs, and it is what we’re building.

Reaching a lab isn’t the hard part. In 2023 an agent called Coscientist read a cloud lab’s documentation and wrote the code for a chromatography run on a caffeine sample, and the lab ran it. The results showed an air bubble injected along with the sample, and someone still had to read what came back. The hard part is everything around the run: the budget, the check, and a record of what was found.

That part works today on your own machine, with research methods plugged in on top; the one from the six-day run is among them. Rented GPUs come next, then labs that take orders through software, then panels of people for questions only people can answer. Each plugs into the same layer, and every result, wherever it ran, lands in the same record under the same rule.

What comes next

Over that record we’re building the belief graph, where verified progress accumulates: every result the units have verified, each tied to the evidence behind it, kept current as new evidence strengthens or overturns it. Every new question starts from it, so a unit begins where the last one stopped. It is also what the next models can learn from, as our thesis argues: results nobody could have taught them, because nobody had them yet.

If it works, one researcher can keep dozens of experiments going by talking, and every result they get back carries its evidence. Each result should also make the next one cheaper to find, and we’ll publish that curve whichever way it goes.

What would change our mind

  • If agents left alone reach as many verified results per dollar as a person and agents together, on the same questions and budgets, steering is a passing phase and we’ve built for the wrong one.
  • If every new place to run experiments needs its own harness rather than plugging into one layer, there is no layer to build.
  • If later research units don’t reach verified results more cheaply than earlier ones, the belief graph isn’t compounding.

Numbers about the six-day run come from our paper and our earlier post. Figure 1 is a model, and figure 2 is two screenshots of a test record with scripted workers. The external sources were checked against their text on 23 October 2026.

Point it at a problem

If you have a hard question and a way to tell a better answer from a worse one, or you want to help build the harness, talk to us.

Sources

Research

  1. Ghosh, Recursive Priced Falsification: Asymmetric Trials with a Forecast-Scored Planner for Autonomous Discovery, October 2026. The source for every number about the six-day run. Our preprint; self-reported
  2. Pensieve, Most trials should lose, October 2026. The run, for a general reader. Our post
  3. Kwa, West, Becker et al., Measuring AI Ability to Complete Long Software Tasks, arXiv:2503.14499, March 2025. The 50% time horizon and its doubling. Preprint
  4. METR, Research Update: Algorithmic vs. Holistic Evaluation, August 2025. Tests passed against changes a maintainer would merge. Research note; small sample
  5. Szymanski, Rendy, Fei, Kumar et al., An autonomous laboratory for the accelerated synthesis of inorganic materials, Nature 624, 86-91, 29 November 2023, as corrected. Peer-reviewed
  6. Szymanski, Rendy, Fei, Kumar et al., Author Correction: An autonomous laboratory for the accelerated synthesis of inorganic materials, Nature 650, E1, 19 January 2026. Correction
  7. Leeman, Liu, Stiles, Lee, Bhatt, Schoop and Palgrave, Challenges in High-Throughput Inorganic Materials Prediction and Autonomous Synthesis, PRX Energy 3, 011002, March 2024. Peer-reviewed
  8. Boiko, MacKnight, Kline and Gomes, Autonomous chemical research with large language models, Nature 624, 570-578, 2023. The cloud-lab run is in the section on the lab’s programming language. Peer-reviewed
  9. Long, Pender, Huang, Santos, Singh et al., ECDSA.Fail: Open Autoresearch for Optimizing Elliptic-Curve Point Addition in Shor’s Algorithm, arXiv:2609.09582, September 2026. The benchmark and its rules. Preprint, from the benchmark’s organisers

Announcements and reports

  1. Anthropic, How AI is transforming work at Anthropic, 2 December 2025. A survey of 132 of its staff and its own usage data. Company report; self-reported
  2. Cognition, Introducing Devin, the first AI software engineer, 12 March 2024. Company announcement
  3. Cognition, Cognition’s acquisition of Windsurf, 14 July 2025. Company announcement
  4. Anthropic, Anthropic acquires Bun as Claude Code reaches $1B milestone, December 2025. Company announcement