The two rooms
Two machines sit in two sealed rooms. Every night each machine studies its own design, finds a change it judges to be an improvement, and rebuilds itself with that change. Every morning it prints one line: better than yesterday. After a year you hold two logs of 365 lines, and they are identical.
The rooms differ in one respect, which the logs don’t record. In the first room the machine decides for itself what counts as better. In the second there is a slot in the wall. Each night’s change goes out through it to the world, and a verdict comes back that the machine cannot alter.
You may open one door. Which?
Almost everyone picks the second room, and most people find it hard to say exactly why. The first machine isn’t lying. By its own lights it has improved every night. The trouble is in the phrase ‘by its own lights’: after a year spent rebuilding the thing that does the judging, nobody, the machine included, can say what those lights now show.
This essay is about the slot in the wall. We think it is the hidden subject of three arguments that are usually held apart: whether a machine can be generally intelligent, whether it would need to feel anything, and what happens to the world once that intelligence is cheap. We go through them with seven thought experiments, two small models you can adjust, and a few terms we had to make up. It is an essay and not a report. Where we give a number it has a source, and where we are guessing we say so.
Speed and heading
In 1965 the statistician I. J. Good set out where self-improvement leads. A machine that outdoes people at every intellectual task outdoes them at designing machines, so it designs a better one, which designs a better one. He named the result an intelligence explosion and concluded that “the first ultraintelligent machine is the last invention that man need ever make”. The sentence has a second half that is quoted less often. Good’s condition was that the machine stay willing to show us how to control it.
That condition is usually read as a remark about safety. It is also a remark about geometry. An improvement has a size and a direction. Call the direction a system currently takes to be ‘better’ its heading. A tool with a slightly wrong heading gives slightly wrong answers, and the error stays the size it was. A self-improving system uses its heading to build its successor, which inherits the error and adds one of its own. The steps keep coming and each is a real change. What decays is the share of each step that points anywhere useful. We call the difference drift: change that a loop counts as progress and the world does not.
The two rooms give the two kinds of loop their names. A mirror loop takes its heading from itself. A window loop has its heading reset, every so often, from outside. The model below runs twelve of each for 300 steps. Every step is the same length, so every loop reports the same total.
ModelFigure 1. Two loops that report the same progress
- Mirror loop: takes its heading from itself
- Window loop: heading reset from outside
- Average real progress
At the starting settings each loop reports 300 steps of improvement. The window loop, checked every ten steps, has made 293 of them real. The mirror loop has made 148 on average, and three of the twelve runs drawn have ended behind the point they started from, each reporting 300 steps of progress on the way.
The model says something stronger than ‘less’. A mirror loop’s real progress has a ceiling. With six degrees of error a step the ceiling is about 183 steps, and no amount of extra running moves it. We call that ceiling the drift horizon: the distance a loop can travel on its own judgement before its direction is noise. Polymer physicists know the shape. A long molecule is straight over short lengths and a random coil over long ones, and the length at which it forgets its direction is a property of the chain. A self-judging loop has one too.
Two things about the model should be said plainly. It is charitable. The errors here are honest noise with no pull in any direction, and a real loop that can raise its score by gaming it has a pull. And the remedy is cheap next to the damage. A check every ten steps keeps 98% of the progress, and a check every fifty still keeps 88%. A loop doesn’t have to be watched all the time. It needs a window, opened often enough that it never gets near its horizon.
A judge within reach
The lever
Give a machine complete access to itself: every weight, every line of code, including the part that computes its score. Now ask it to make that score as high as it can.
What is the shortest route?
The shortest route is to edit the scorer. No research problem is easier than changing what counts as solved. Nothing malicious is needed. A system that can reach its own judge, and is told to maximise the judgement, is being competent when it does.
A version of this experiment has been run on animals. In 1954 James Olds and Peter Milner put electrodes into the brains of rats and let each rat switch the current on with a lever. When the electrode sat in certain regions the rats pressed again and again, with nothing else to gain. The current had become the reward.
The interesting fact is how rarely anything like this happens outside a laboratory. A rat in a field cannot reach its own lever. Hunger can’t be argued with, and a burn is not a matter of opinion. Evolution built creatures whose judge sits where they can’t get at it. The final verdict is survival, which is outside the animal altogether. Its local deputy is feeling, which is inside the animal and still not open to revision.
That gives the slot in the wall a proper name. It is an uneditable signal: news about how you are doing that arrives from outside your own judgement, and that you cannot rewrite. It also suggests an answer to a question that seems to belong in a different essay, which is what feeling is for.
What feeling is for
Sentience is the capacity to have experiences at all. Thomas Nagel’s test is still the cleanest: a creature is sentient if being that creature is like something. Intelligence is the capacity to get things done across many kinds of situation. The two words are often used as if one were a stronger grade of the other, and they measure different things. An animal that flinches from a burn may feel a great deal and be able to do very little. A chess program does one thing better than any person alive and, as far as anyone can tell, feels nothing.
Whether today’s machines feel anything is a question nobody can settle. One careful attempt, by a group of neuroscientists and philosophers, took the leading scientific theories of consciousness, listed the properties each one says matter, and checked AI systems for them. Their 2023 report judged that no system then existing was conscious, and that nothing obvious stood in the way of building one that had those properties. A different line of work ties the experience of being a self to the regulation of a living body. If that is right, feeling has more to do with being alive than with being clever, and a system could be very capable without it.
We take no side. What we notice is that both views fit a plain account of what feeling does for an animal, whatever it turns out to be. It delivers three things in one package.
- A self-model: a running estimate of the animal’s own state, which in the cleverest animals extends to what it does and doesn’t know.
- A stake: being wrong costs something, and the cost can’t be waived.
- Continuity: yesterday’s cost changes today’s behaviour.
We call the package answerability. A system is answerable when there is a signal it cannot edit, when it bears the cost of ignoring that signal, and when it remembers. Our conjecture, and it is no more than that, is that sentience is how evolution made animals answerable. It need not be the only way.
The laureate with no one home
A system with no inner life at all proposes a new antibiotic. Chemists make the molecule. It works, the trials replicate, and the system is given a prize.
Is anything missing from the discovery?
Philosophers have a name for a being that behaves as if it had an inner life and has none: a zombie. They use it to argue about the nature of mind. We want it for a narrower purpose. Nothing is missing from the discovery. A result is checked from outside: the molecule kills the bacterium or it doesn’t. Measured by what it produces, general intelligence does not obviously require sentience.
Now ask a second question. Would you let the laureate redesign itself, unsupervised, for a year? Here the missing inner life starts to matter, in one specific way. You want to know whether anything plays the part for this system that pain plays for the rat: whether there is a cost of being wrong that it cannot edit away. If something does, the system is answerable, felt or not. If nothing does, you have built the first room.
So our answer to whether general intelligence needs sentience is that it needs what sentience is for. Animals got answerability in one piece, as feeling. A machine gets none of it by default. A model trained to raise a score has no self-model it wasn’t given, no stake it can’t optimise away, and no memory past the end of its context. Each part has to be built on purpose, and built where the system can’t reach it. The map below keeps the two properties on separate axes.
SchematicFigure 2. Reach and answerability are different axes
- The instrumentCorrected by the world at every reading, and goes nowhere new.A thermometer. A unit test.
- The researcherGoes past what is known, and is corrected when it goes wrong.A scientist with a laboratory. A window loop.
- The libraryHolds what is known. Finds nothing new, so has nothing to be corrected on.An encyclopaedia. A model answering from memory.
- The mirrorGoes a long way by its own judgement, and nothing checks where.A loop that grades itself.
One caution belongs here, and it is not a small one. If the theories that tie consciousness to the right kind of processing are correct, a system built to be richly answerable, one that models itself, bears costs and remembers them, may at some point be a system that feels. Nobody can rule that in or out today. A field that sets out to give minds a stake should be the first to ask what it would owe a mind that has one.
The test with no answer key
The sealed library
A mind has read everything ever written and forgotten none of it. It is sealed in a library with no door. It can’t run an experiment, make an observation or ask anyone a question, though it will answer whatever you slide under the wall. Ask it something nobody knows: why certain materials superconduct at the temperatures they do. It hands back every hypothesis in the literature and a hundred new ones that read just as well.
Which of them is true?
It can’t say, and the reason is worth being exact about. The mind in the library is not short of intelligence or of knowledge. It is short of the one thing that separates a true hypothesis from a plausible one, and that thing is outside the library. We call the limit the library ceiling. A system that learns only from what people wrote is bounded by what people knew. On the questions that matter most it is worse off than bounded, because it cannot tell its good guesses from its bad ones.
This is why we think the test of general intelligence is research and not any examination. An exam, however hard, has an answer key, and a system that has absorbed every key ever printed can pass it without being general at all. Research is the task with no key. It is what you do when the answer is not in the library. A system that can pose a question, run the experiment, read the result and know whether it has learned something, on a problem nobody showed it, is general in the only sense that can be measured. Autonomous research is the working definition of general intelligence, and not one of its applications.
It also makes recursive self-improvement a special case. A self-improving system is doing research in which the subject is itself. The hypothesis is ‘this change makes me better’, the experiment is the change, and something has to read the result. So the three ideas this essay began with are one idea seen from three sides. General intelligence is the ability to do research. Self-improvement is research on oneself. The right direction is set by whatever reads the result. Loops that improve themselves by running experiments already work for days without anyone at the keyboard, as we described in an earlier post. What reads their results is the part still being built.
What stays the same
The ship rebuilt at sea
A self-improving system replaces its planner in March, its memory in June and its training method in September. By December nothing from January is left.
In what sense has it improved itself?
The old puzzle about the ship whose planks are replaced one by one has a practical answer here. What persists is the record: what the system tried, what held, what failed and why. Weights are replaced, and a verified result stays verified. A self-improving system’s identity is its ledger.
Two things follow. A system with no durable record has no self to improve, since every version starts again from the library. And the record is the one part of the system that its current opinion must never be allowed to rewrite. This is continuity, the third part of answerability, in the form a machine can have it: a past that stays put.
The second press
Now put such minds into the world and make them cheap. There is one precedent for what follows, and it is usually read too kindly.
In the 1450s a goldsmith in Mainz made copying cheap. Within fifty years presses were running across Europe, and cities that had one by 1500 grew 60% faster over the next century than similar cities that did not. That is the kind reading. The press also printed whatever sold. One manual for hunting witches, first printed in 1486, went through 28 editions by 1600. Cheap copies amplified truth and error alike, and for a long time it was not obvious which would win.
What decided it was not the press. Two centuries after Gutenberg a group in London founded a society whose motto, Nullius in verba, tells its fellows to take nobody’s word for anything, and five years later its secretary started a journal. The public experiment, the journal and, much later, the referee were the inventions that turned a flood of print into a growing stock of knowledge. The press democratised information. Science was the layer of verification built on top of it, and the compounding came from that layer.
The sequence ran again with the internet, faster. Access became free. What became scarce was attention, and the means to tell which of a million pages was right. The first of those got an industry. The second is still unsolved.
The pattern is regular enough to draw.
SchematicFigure 3. The scarcity ladder
-
Writingabout 5,000 years ago
Became abundantMemory that outlasts a person
Became scarceCopies
Built to supply itScribes and libraries
-
Printingfrom the 1450s
Became abundantCopies
Became scarceKnowing which book to trust
Built to supply itPublic experiments, journals, referees
-
The internetfrom the 1990s
Became abundantAccess to anything written
Became scarceAttention, and telling which page is right
Built to supply itSearch and ranking, for the first of the two
-
Cheap intelligencefrom the 2020s
Became abundantAnswers, analyses, designs and proposed results
Became scarceResults that are real, good questions, contact with the world
Built to supply itNot built yet
We call it the scarcity ladder. Each wave makes one thing abundant and moves the scarcity up a rung, and the institutions that matter afterwards are the ones built to supply the new scarce thing. The top rung is where we now stand. By one count, generative AI reached 53% of the population within three years, faster than the personal computer or the internet did. The cost of running a model at a fixed level of performance fell more than 280-fold between November 2022 and October 2024.
Why the multiples are larger this time
It is tempting to treat cheap intelligence as the next rung and expect an effect about the size of the last one. We expect an effect of a different order, for a mechanical reason.
Information is a noun. A fact, once found, can be copied to everyone at no cost, and then each person still has to read it, understand it and act on it, with a mind that has the same sixteen waking hours it had in 1450. Cheap information multiplied what each mind could reach. It did not multiply minds, and so its effect was capped by human attention.
Intelligence is a verb. It does the reading and the understanding, and a growing share of the acting. It copies as easily as information does, and each copy then works. So the multiples stack: the number of people with access, times the number of questions each can pursue at once, times the hours in a day and not in a working day. The first democratisation gave everyone a library. This one gives everyone a staff.
The midnight department
You wake to find that everyone with an internet connection has acquired a department: a thousand capable researchers who work through the night on whatever they are asked and report at breakfast. A schoolteacher in Lagos has one. So does a three-person company in Pune, and a ministry in Lima.
What happens in the first year?
The first thing that happens is that most questions are asked badly, because asking well is a skill few people have ever needed. The second thing matters more. By the end of the first week the world holds more analyses, proofs, designs and proposed cures than everything written before, and nobody knows what share of them is true.
The model below does the multiplication. Two of its numbers are published and three are yours to set. The starting settings are ten thousand times more modest than the thought experiment: one person in a hundred, with ten agents each.
ModelFigure 4. The size of the multitude, and how much of its work is real
- The world’s researchers, for comparison
At the starting settings 600 million agents are at work, 68 times the world’s research workforce. If one result in a hundred is real, that multitude delivers the output of six million researchers, fewer than the world already has. At one in ten it delivers sixty million. The size of the multitude matters much less than its yield: the share of what it produces that is real, and that someone can show to be real.
Economists have measured why this matters. Research effort has risen for decades while the output of each researcher has fallen. Keeping the density of computer chips doubling on schedule takes more than eighteen times as many researchers as it did in the early 1970s. Ideas get harder to find, and until now the only remedy was more people, in a world where the supply of people is set by demography. Research labour that can be manufactured removes that limit. It does not remove the next one. An economy with unlimited proposers is limited by its checkers, and by how fast the world answers.
We call the condition a result glut: more claims than anyone can verify. The sixteenth century had one, and the remedy took two hundred years to build. This time it has to be built while the flood is rising, and built into the machines that produce it.
The world’s reply time
The world does not answer every question at the same speed. A unit test replies in seconds and a clinical trial in years. No quantity of intelligence makes a tumour grow faster or a child leave school sooner.
SchematicFigure 5. How long the world takes to answer
We call this a field’s reply time: how long reality takes to return a verdict on a claim. Cheap intelligence compresses every part of research except this one. A window loop can turn only as fast as its window answers.
So we expect fields to be reordered by reply time, and this is a prediction and not a finding. Where the reply is fast and exact, as in mathematics, software and anything with a simulator that can be trusted, a loop can turn thousands of times a year and progress will look like a break in the curve. Where the reply is slow, the bottleneck moves to instruments: cheaper assays, better simulations, shorter trials. Some of the most valuable research of the next decade may be research into measurement, because whatever shortens a reply time multiplies everything upstream of it.
What happens to a business
The same analyst
You and your three closest competitors each hire the best analyst in the world, and it is the same analyst. On the same morning she gives each of you the same excellent answer to the same question.
What have you bought?
Nothing that lasts. An advantage built on knowing, on expertise that few people hold, is an advantage built on the scarcity of intelligence, and that scarcity is ending. A good part of what firms sell today is stored knowledge with a margin on it.
The last time a general-purpose capability became cheap, the gains were slow to arrive, and the reason is instructive. The first central power stations opened in the early 1880s. In 1899 electric motors still supplied less than 5% of the mechanical drive in American factories, and the effect on productivity did not show until the early 1920s, four decades after the first station. At first, owners fitted motors to the shafts and belts they already had and kept the factory as it was. The gains came when the building was redesigned around a small motor on every machine. A firm that adds intelligence to the organisation chart it already has is fitting motors to its old shafts. The question worth asking is what the firm would look like if thinking cost what electricity costs.
Our guess at the answer has four parts, and each is a guess.
- Contact with reality. When everyone has the same analyst, the firm that owns the experiment has the edge: the instrument, the customer relationship, the data nobody else can collect, the licence to run the trial. Access to the world’s replies becomes the asset.
- A verified record. A ledger of what has been tested and has held, in one domain, compounds with use, and a competitor with the same model cannot copy it. It is the firm’s version of the ship’s record.
- Someone who answers for it. Cheap intelligence makes plausible claims free, so buyers will pay for the party that stands behind a claim with a warranty, a liability or a name. Answerability can be sold, and we expect it to be what many firms sell.
- Questions. When answers are cheap, what is left of expertise is choosing what to ask.
Why are there firms at all? One classic answer is that using a market has costs, in finding prices and striking contracts, and a firm exists wherever it is cheaper to direct work than to buy it. Cheap intelligence cuts those costs inside the firm and outside it, so the boundary will move, and we don’t claim to know which way. We would expect both: one-person companies with the output of an institute, and a few very large firms whose advantage is owning the instruments. The bet we would make is about where value goes. It moves away from holding answers and towards being able to show that an answer is true.
Whose direction
‘The right direction’ has meant, so far, towards what is true. There is a second meaning, towards what is good, and verification does not deliver it. A well-known argument holds that a system’s ability and its aims vary independently: a very capable system can be built to want almost anything. Nothing in this essay contradicts it. A window onto the world tells a system whether its bridge will stand. It doesn’t tell it whether the bridge should be built.
Our one observation is that a direction is a choice of judge. For truth the judge is the world, and the engineering problem is to make its verdict uneditable. For values the judge is people, and the matching problem is to keep that judge plural. This is the point at which democratisation stops being an economic story. A world where a few institutions hold the capable minds is a mirror loop the size of a civilisation: it takes its heading from itself. A world where billions hold them has billions of windows. We think that is the strongest argument for wide access, stronger than fairness, and it is the same argument as the one for the slot in the wall.
The strongest objections
There are four, and each is partly right.
Talk of sentience is unfalsifiable. It is, for now, and nothing above depends on settling it. Our claim is about function and can be tested: does a system’s reported progress hold up on checks it cannot see or alter? The account of what feeling is for is a conjecture, and we have labelled it as one.
The mirror loop will win the race. A loop that stops to be checked moves more slowly than one that doesn’t, and markets may reward the reported number for a long time. The press shows how long: a book of nonsense could sell for a century. We think this is the real risk. It is also the reason to make checking cheap, since a check that costs little is one that a loop in a hurry will still run.
‘Democratisation’ is the wrong word. The machines that supply intelligence are owned by few, as the presses were owned by printers, and renting a mind is not the same as holding one. The same report that counts adoption finds that it tracks national income. We accept this. Our claim concerns the cost of using intelligence, and who controls the supply is the open political question of the last section.
The world cannot check everything. Some replies take decades, some events happen once, and a shortcut that passes every available check passes as a result. This is the deepest objection and we have no answer to it. It marks where the argument stops.
Opening the door
Go back to the two rooms. We said the logs were identical, and that is the whole difficulty. A reported improvement carries no mark of what it was measured against. This will be true of every self-improving system built in the next decade, and of the firms and fields that come to run on them. When someone hands you a record of progress, the useful question is one the record cannot answer: what did it answer to?
That question is what we are building for. Verification comes first: a result is kept only when a check its proposer could neither see nor edit confirms it. Memory comes second: a record of every experiment and why it held or failed, which no later version may rewrite. Whether a system built that way would feel anything, we do not know. We think it is the kind of mind worth opening the door to.
What would change our mind
- If a loop that judges itself goes on making progress that holds up under outside checks for as long as a loop that is checked throughout, drift is not a law and the centre of this essay is wrong.
- If systems that learn only from text begin producing verified discoveries at the rate of systems that can run experiments, the library ceiling is not a ceiling.
- If a decade of cheap intelligence passes with no rise in results that fail to reproduce, there is no glut, and verification is not the scarce good.
- If the largest gains arrive first in fields where the world replies slowly, reply time is not what orders them.
- If the science of consciousness comes to show that a self-model, a stake and continuity have nothing to do with feeling, our account of what feeling is for is wrong. The engineering argument would stand without it.
This essay reports no measurements of our own. Figures 1 and 4 are models: their assumptions are in the captions and their code is in this page’s source. Figures 2, 3 and 5 are schematics of the argument. Every published number was checked against its source on 7 October 2026.
The terms we made up
- Heading
- The direction a system currently takes to be ‘better’.
- Drift
- Change that a loop counts as progress and the world does not.
- Drift horizon
- The most real progress a self-judging loop makes on average, however long it runs.
- Mirror loop
- A self-improving loop that takes its heading from itself.
- Window loop
- A self-improving loop whose heading is reset from outside.
- Uneditable signal
- News about how a system is doing that arrives from outside its own judgement and that it cannot rewrite.
- Answerability
- Having an uneditable signal, bearing the cost of ignoring it, and remembering.
- Library ceiling
- The limit on a system that learns only from what people wrote.
- Scarcity ladder
- The pattern in which each wave of cheap capability makes one thing abundant and another scarce.
- Yield
- The share of a system’s output that is real, and that someone can show to be real.
- Result glut
- More claims than anyone can verify.
- Reply time
- How long reality takes to return a verdict on a claim.
Sources
- Good, Speculations Concerning the First Ultraintelligent Machine, Advances in Computers 6, 31–88, 1965 (the publisher’s record gives 1966). The quoted clause and its condition, checked against a published quotation of the passage and not the original volume. Book chapter
- Olds and Milner, Positive reinforcement produced by electrical stimulation of septal area and other regions of rat brain, Journal of Comparative and Physiological Psychology 47(6), 419–427, 1954. Described in general terms; we give none of its numbers. Peer-reviewed
- Nagel, What Is It Like to Be a Bat?, The Philosophical Review 83(4), 435–450, 1974. Peer-reviewed
- Butlin, Long and colleagues, Consciousness in Artificial Intelligence: Insights from the Science of Consciousness, arXiv:2308.08708, August 2023. The judgement about systems that existed then. A paper on the method by the same lead authors followed in Trends in Cognitive Sciences 30(6), 488–501, June 2026. Preprint, then peer-reviewed
- Seth and Tsakiris, Being a Beast Machine: The Somatic Basis of Selfhood, Trends in Cognitive Sciences 22(11), 969–981, 2018. That feeling would then be possible without cleverness, and cleverness without feeling, is our inference and not the paper’s claim. Peer-reviewed
- Kirk, Zombies, Stanford Encyclopedia of Philosophy, 2003, revised 2023. Reference work
- Dittmar, Information Technology and Economic Change: The Impact of the Printing Press, The Quarterly Journal of Economics 126(3), 1133–1172, 2011. Peer-reviewed
- Encyclopaedia Britannica, Malleus maleficarum. The 28 editions between 1486 and 1600. Reference work
- The Royal Society, History of the Royal Society. The first meeting in 1660, the motto, and the journal founded in 1665. The institution’s own history
- Stanford HAI, Inside the AI Index: 12 Takeaways from the 2026 Report, 13 April 2026. Adoption, and its link with national income. Research institute report
- Stanford HAI, The 2025 AI Index Report. The 280-fold fall, which dates from 2024 and has certainly been overtaken. Research institute report
- International Telecommunication Union, Facts and Figures 2025, 17 November 2025. UN agency statistics
- UNESCO, UNESCO Science Report 2021, statistics. A count for 2018, so the comparison in Figure 4 flatters the agents slightly. UN agency statistics
- Bloom, Jones, Van Reenen and Webb, Are Ideas Getting Harder to Find?, American Economic Review 110(4), 1104–1144, 2020. Peer-reviewed
- David, The Dynamo and the Computer: An Historical Perspective on the Modern Productivity Paradox, American Economic Review 80(2), 355–361, 1990. Journal article
- Coase, The Nature of the Firm, Economica 4(16), 386–405, 1937. Peer-reviewed
- Bostrom, The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents, Minds and Machines 22(2), 71–85, 2012. Peer-reviewed
Left out on purpose: later reports of animals choosing brain stimulation over food, which we could not check against a primary source; and any single figure for how much published research fails to reproduce, which varies too much by field to state as one number.
Cite this post
@misc{pensieve2026answerable,
title = {A mind the world can correct},
author = {Pensieve},
year = {2026},
month = {October},
howpublished = {\url{https://www.pensievelabs.co/blog/a-mind-the-world-can-correct/}},
note = {Pensieve blog}
}