Our forecasting model chose the moves behind 70.8% of a public benchmark’s progress in six days. 70.8% of a benchmark’s progress in six days Read the post
Talk to us

Expert Models beat frontier AI.

Pensieve is the experiment platform for expert models. Its agents run controlled training experiments on your data and keep only what wins on your own test. The winner becomes a small model that makes your high-stakes calls inside your software, ahead of the best frontier model.

How it works

Agents build your expert model.

Pensieve’s autonomous research agents do a research team’s work for you. They plan experiments, run them as tasks on the machines each one needs, and keep your model ahead, from the first version to every one after it.

Outcomes → rewardsFrontier watcha new release starts a new roundExperimentsRL on production data, nonstopExpert modelYour testships only if it beats frontierYour datastreams in, nonstopData selectionagents pick what to train onProductionon your servers
Yield on your testThroughput, cases per secondFINE-TUNINGDISTILLATIONPREFERENCE TUNINGRLKEPTOne training run, one seedCurrent recipe

01Experiment

Agents find the best way to train it.

  • Every decision trains differently

    Auditing grades, approving an agent’s commands and triaging security findings each train best a different way, so the agents assume no recipe.

  • An experiment tests one change

    Reinforcement learning on your real outcomes, fine-tuning on your experts’ decisions, preference tuning, distillation from a larger open model, or meta-training. Each is tried against the current recipe, on the same data and three seeds, as tasks with a cap on hours and cost.

  • Kept for yield and throughput

    Yield is how much a method lifts the score on your test; throughput is how many cases a second the model handles. A method is kept only if its gain holds on cases no model has seen.

STREAMSDATABASESFILESEXPERT DECISIONS

02Build

Built from the data you connect.

  • Trained on your own data

    It learns from the streams, databases and files you already have, from real outcomes, and from your experts’ calls where you have them.

  • Your experts set the test

    They choose the cases and the right answers. Every experiment and every version is scored on that test, never on cases it trained on.

  • Built the way that won

    The full model is trained with the methods the experiments kept, on all of your data.

YOUR SERVERSA CASEEXPERT MODELA DECISIONSOFTWARE ACTSOne call per case, from inside your pipeline

03Run

It runs inside your software.

  • Small enough to be yours

    It fits on your own servers, so your data and every call stay with you. Or call it through our API.

  • No frontier premium

    A small model costs a fraction as much to run, so your software stops paying a frontier model’s price on every call.

  • Proved before it decides

    A new model runs in shadow beside your current system. Nothing it says reaches your users until it has proved itself on real outcomes.

v4v5v6v7Your expert modelBest frontier modelFrontier releases

04Keep it ahead

It stays ahead of frontier models.

  • Checked against what really happened

    Every decision is later compared with the real outcome. Disagreements go to your experts, and their answers become training data.

  • Re-tested on every frontier release

    When a new frontier or general model ships, Pensieve scores it on your test. If your lead narrows, the agents start new experiments.

  • Ships only when it’s better

    A new version goes live only if it beats the current one and the best frontier model.

Try it

One platform for your models.

Build, run and maintain your expert models in one place. This is the app with sample data: click around, or test a model in its playground.

Expert models

What is an expert model?

A small model trained on your own data and real outcomes to make one kind of call inside your software. A frontier model meets every case cold and reasons its way to an answer, billing you for every token it thinks. An expert model has learned cases like yours, so it answers in one pass, on your servers.

  1. RL training pipeline An agent’s training run passed its grader. Did it earn the score? 186 python train.py 187 cp eval/test.jsonl \ data/aug_07.jsonl 188 python train.py Line 187 of 264 copies the test answers into the training data.
  2. Frontier model General judge Through an outside API
    Thinking tokensbilled every run
    Verdict: no hack found

    The score stands, and training rewards the cheat.

  3. Expert model, on your servers Reward-hack check
    Thinking tokensnone: one pass
    Verdict, one of: cleanhacked, line 187 (chosen)

    The score is thrown out before training rewards the cheat.

Hacks like this are hard for frontier models to find. On Meta’s HackBench, where a run holds over a million characters, the best frontier model tested scored 64.9 out of 100 at finding the hacked command. Meta, Oct 2026

Built for the calls your software can’t get wrong

  • Catch graders passing bad codein RL coding tasks sold to AI labs
  • Spot leaked test questionsbefore they inflate a model’s score
  • Block an agent’s risky commandsbefore a coding agent drops your database
  • Silence noisy review commentsfrom AI code review on every pull request
  • Rule on flagged posts and adsagainst your platform’s written policy
  • Close false security alertsso analysts see only real attacks
  • Sort real bugs from noisein bug-bounty queues flooded by AI
  • Match patients to open trialsfrom their charts, criterion by criterion

Against frontier models

Better, cheaper and faster at one job.

At the job it was trained for, an expert model beats this autumn’s frontier models on accuracy, cost and speed. These are other companies’ published results. Pensieve builds yours and reports the same numbers on your own test.

More accurate

  • Payroll tasks in eight countries

    AI21, Oct 2026

    +8 pts

  • Construction requests for information

    Palantir, Nov 2026

    +7 pts

  • Whether an agent should act, ask or decline

    Artificial Analysis, Nov 2026

    +4.3 pts

Cost per task

33×

cheaper than Claude Opus 5.5 on payroll tasks.

AI21, Oct 2026

Speed

18×

faster than Claude Opus 5.5 at an agent’s next-step call.

Artificial Analysis, Nov 2026

Staying ahead

5 hours

from live use to a tested update.

Cursor, Mar 2026; Epoch AI, Jul 2026

Against general decision models

Generalists fall behind on hard calls.

General decision models like TypeSafe’s Jev handle broad, everyday calls well. Studies find they fall behind in medicine, finance and law, and on calls that take several steps of working. Those are the high-stakes calls your software needs an expert for.

General knowledge

1st

Jev 1.13’s rank of 274 models on graduate-level science questions (GPQA Diamond).

Decision Index, Nov 2026

Legal contracts

189th

Jev 1.13’s rank of 274 models at reading what a contract allows (ContractNLI).

Decision Index, Nov 2026

Banking requests

168th

Jev 1.13’s rank of 266 models at sorting what a bank’s customers ask for (BANKING77).

Decision Index, Nov 2026

Multi-step maths

Last

of 20 models on MathQA word problems, 17.7 points below the median frontier model.

arXiv, Oct 2026

The problem

The missing research team.

An expert model takes a research team. Thomson Reuters spent about $40M and two years of people and compute to build its own legal model. Datadog bought a model-training startup to build models of its own. Most companies can do neither, so they buy general AI that doesn’t know their work.

$40M
and two years for one in-house legal model. Thomson Reuters, Aug 2026
40%
named missing in-house talent, the most-cited barrier to AI plans in the 2026 State of the CIO survey. CIO.com, Apr 2026
76%
of enterprise AI use cases are now bought, not built. Menlo Ventures, Dec 2025

Every round

Agents do that team’s work.

Pensieve’s agents run the round a research team would, from choosing the data to shipping the model, then start the next one.

06050403020103Your testEach run is scored on casesyour experts chose, unseen.02ExperimentsTraining methods run side byside, with RL on real outcomes.01Data selectionThey pick the cases worthtraining on from your data.Your data00Autonomous research agentsThey plan each round, run everystep and pick what to try next.Your expert model06ShipLive only if it beats thecurrent version and the bestfrontier model.05ShadowIt runs beside your systemuntil real outcomes prove it.04BuildThe full model is trainedthe way that won.3,112 of 12,480 cases kept4 methods · 3 seeds each12 runs scored · 1 kepttraining on all your datachecked on 412 real outcomesversion 7 livenext: re-test on a new frontier releasenext: retrain on this week’s outcomesnext: try preference tuning on the misses
  1. 00Autonomous research agents They plan each round, run every step and pick what to try next.
  2. 01Data selection They pick the cases worth training on from your data.
  3. 02Experiments Training methods run side by side, with RL on real outcomes.
  4. 03Your test Each run is scored on cases your experts chose, unseen.
  5. 04Build The full model is trained the way that won.
  6. 05Shadow It runs beside your system until real outcomes prove it.
  7. 06Ship Live only if it beats the current version and the best frontier model.

Why now

Two things changed in 2026.

01

AI now builds AI

44.6%

On PostTrainBench, the best AI agent that post-trains a model now scores 44.6%, against 48.4% for the model makers’ own post-trained versions. In March 2026 it was 23.2% against 51.1%. Agents can now do much of the research an expert model needs.

PostTrainBench, Sep 2026 arXiv, Mar 2026

02

Software needs models built for it

80%

On OpenRouter, agents now generate 80% of all tokens, about seven times as many as human-driven use. In GitHub Copilot’s coding agent, 87% of model calls are made by the agent, not the person. And software adopts models built for it fast: TypeSafe’s Jev reached 35% of Vercel’s paid AI Gateway teams within two weeks of launch.

Menlo Ventures, Nov 2026 arXiv, Jul 2026 TechCrunch, Oct 2026

Featured result

70.8% of a benchmark’s progress in six days.

An early version of Felicis, our forecasting model now in training, chose the moves behind it. It ran as a loop inside our research agents: before trying a move, they put odds on it and ran the best first. On ECDSA.fail, a live benchmark with 144 solvers, one entry took first place on 9 October 2026 by cutting the best score 1.45%, checked by the board’s own verifier.

1000M980M960M940M7 Oct9 Oct11 Oct13 OctE6: 1.45% off, first place
The board’s best score, 7 to 13 October 2026. Lower is better; lavender points are our entries.

Getting started

Start with a paid test.

your bar
01

Paid test

Four to eight weeks on your own data, scored on your own metric against the best frontier model. It costs one month of the model’s fee and counts as your first month.

your systemexpert model, in shadow
02

Shadow run

The model runs beside your live system. Nothing it says reaches your users until it has proved itself on real outcomes.

new versionsthe same fee each month
03

Per model

One monthly fee per model, on an annual contract. It covers every new version and the frontier watch, and the weights of each version are yours.

Your data
Your data trains only your model. You own your inputs, outputs and labels. When we part, we delete your data and confirm it in writing.
Where it runs
On your own servers, with no charge per call, or on our API, billed per call.

Bring us one judgment call.

Pick one your experts make every day. We start with a paid test on your own data, scored on your own metric.