Any system that adapts how AI work is executed has to decide where the adapting happens.

The tempting answer is: on every request. Put a model in front of the model, let it read each request, and let it choose the configuration that should handle it. It is flexible and easy to demonstrate. It also means paying for an extra decision on every request—in latency, in cost, and in a choice that can’t be reproduced exactly, because the thing making it is itself stochastic.

The opposite answer is to decide once. Pick a model and settings, ship them, and revisit them when someone complains. That is fast and predictable, and it goes stale the moment prices, models, or traffic change.

We think the answer is to separate the two jobs.

The slow loop learns

The slow loop is where research happens. It observes outcomes, proposes interventions, runs controlled experiments against a baseline, and decides what the evidence supports. It can afford to be expensive per decision because it makes few decisions: it works on a scale of hours and days, not milliseconds. It can use large models, repeated trials, and careful statistics. Nothing it does touches a live request directly.

The fast loop serves

The fast loop sits in the request path. For every request, it applies the current validated policy: which model and provider, which inference settings, how context is prepared, and what to do when something fails.

We hold it to a strict definition of deterministic. Given the same policy version, the same task features, the same execution state, and the same environment, it makes the same decision. A small learned predictor is allowed inside that, as long as its behavior is fixed by the policy version.

That definition is narrower than it sounds. It doesn’t make the model’s outputs deterministic—hosted models aren’t. And it doesn’t make the decision fast by itself. The decision path has to be measured on its own, separately from the model call and from total task time.

The same shape, animated on our home page: requests follow the live policy while experiments run in the background, and a validated result updates the policy.

The product is the boundary

What connects the loops is a compiler: the step that turns what the slow loop learned into something compact enough for the fast loop to execute. That is where most of the hard questions live. How much of an expensive policy’s value survives a strict overhead budget? Which request features are worth computing at runtime? When the compiled policy has to choose between two configurations, how costly is a mistake?

That last question matters more than it looks. Confusing two nearly equivalent configurations costs almost nothing. Choosing one that fails the task costs the whole task. A compiled policy should be judged by the consequences of its errors, not by how often it agrees with the expensive policy it replaced.

What the split buys

  • Every runtime decision traces to a policy version, and every policy version to the evidence that justified it.
  • Research can’t break production. Experiments run on isolated replays and controlled trials, and changes are released gradually.
  • Rollback is a version change, not an emergency.
  • The cost of learning is paid once per decision, not once per request.

What it costs

The fast loop is only as current as the last validated update. So the system has to notice when the world moves—new traffic, a new model, a price change, a provider changing behavior behind the same version string—and reopen the question. In this design, drift detection isn’t an optional extra. It’s what keeps a deterministic runtime from becoming a stale one.

If this sounds like training and inference, it should. The slow loop is training: expensive, offline, and evaluated carefully. The fast loop is inference: cheap, online, and fixed until the next release. What’s being trained isn’t a model’s weights but the policy that decides how models are used.

What would change our mind

  • If the best execution decision depends on signals that can only be computed by reading each request with a large model, a model in the request path may earn its cost.
  • If compiled policies lose most of the slow loop’s value under a realistic latency budget, the boundary is in the wrong place.
  • If workloads change faster than the slow loop can validate updates, the runtime will spend too long on stale policies.

This post describes an architecture, not a result, and contains no measurements. When we have them, we’ll report the decision path’s latency separately from model latency.