A familiar claim: “We cut your inference bill by 40%.” The evidence is usually two numbers—last quarter’s bill and this quarter’s.
A lot happens between those two numbers. Providers change their prices. New models appear. Traffic grows in some workflows and shrinks in others. Someone rewrites a prompt, adds a tool, changes a retry policy. The bill falls. Which of those things made it fall?
A before-and-after comparison can’t say. What the claim needs is a counterfactual: what would this quarter’s work have cost if nothing had been optimized?
Buildings solved this first
Energy efficiency has faced the same problem for decades. A retrofit is installed and the energy bill drops—but the winter was also milder, and the building was half empty. Measurement and verification practice, codified in protocols such as IPMVP, answers with an adjusted baseline: estimate what the building would have used under this period’s weather and occupancy without the retrofit, then measure savings against that, not against last year’s bill.
Inference needs the same discipline. The adjustments are different; the idea carries over directly.
Building the counterfactual
An honest estimate of savings separates at least four things:
- Price. Reprice the baseline configuration at current prices, including its caching behavior.
- Volume. Normalize for how much work there was.
- Mix. Account for which workflows grew and which shrank. An optimizer that happens to see more of the easy work will look better than it is.
- The intervention. What remains, with its uncertainty, is the effect of the change.
Some interactions can’t be separated cleanly. Those should be labeled, not absorbed into the headline.
Three ways to get a baseline
- A control group. Keep a random share of tasks on the baseline configuration. It’s the strongest evidence available, and it has a cost: those tasks don’t get the improvement.
- Replay. Run the baseline on recorded traffic after the fact. Cheaper, but valid only when side effects are isolated—and it should be labeled as replay, not as live measurement.
- Off-policy estimation. Estimate the baseline from logged decisions. It works only if the system recorded how likely each decision was when it made it, which has to be designed in from the start.
Savings at lower quality aren’t savings
A cheaper configuration that does worse work hasn’t saved anything; it has moved the cost somewhere else. So a savings claim also needs a quality claim: that the new configuration is no worse than the reference, within a stated tolerance, over a stated volume of work—with an interval, not a point estimate.
Questions to ask about any savings claim
- What was the baseline, and was it repriced at current prices?
- How were changes in volume and mix separated from the effect?
- Was the baseline a control group, a replay, or an estimate?
- What is the interval?
- Were failed and stopped tasks counted?
- Who measured it: an independent party, the vendor, or the customer?
We intend to hold our own claims to the same list.
What would change our mind
- When prices, volume, and mix genuinely stay the same over the comparison period, before-and-after is a fair estimate. That’s rare, and the assumption should be stated.
This post contains no measurements.