Part 1 built the machine and Part 2 taught it to allocate honestly. Then the users arrived with the only question that ever mattered to them: "Is my test done? Can I ship the winner?"
It's a fair question with a surprisingly slippery answer. A bandit never finishes - it just keeps reallocating. Two equivalent variants will split the probability of being best forever. This final part is about turning "done" into a statistical verdict - using expected regret rather than confidence - and about all the product machinery that question dragged in: promoting winners and letting a finished experiment graduate into a plain, boring template.
The Question P(best) Can't Answer
The natural first instinct is a confidence threshold: declare the test done when the leader's P(best) crosses, say, 95%. For slots with a genuine winner, that works fine - evidence accumulates, the posterior sharpens, P(best) marches to ~100%, done.
But recall the ghost that haunted Part 2: two equivalent variants never resolve. Their P(best) hovers around 50/50, drifting with noise, indefinitely. Under a confidence-threshold rule, that test runs forever - eternally "still deciding" - even though, and this is the key realization, there is nothing left to decide. If two subject lines truly perform the same, committing to either one costs you nothing. The test is done; the confidence rule just can't see it.
The question the user is actually asking isn't "are you sure which one is better?" It's "what would it cost me to commit right now?" Those are different questions, and the second one has a name: expected regret.
Stopping on Regret, Not Confidence
For each slot, the optimizer computes the expected relative regret of committing to the current leader:
$$\text{regret}_{\text{rel}} = \frac{\mathbb{E}\big[(\max_i \theta_i - \theta_{\text{leader}})_+\big]}{\mathbb{E}[\theta_{\text{leader}}]}$$
In words: across the posterior draws, how much reward rate do you sacrifice, on average, in the worlds where the leader isn't actually the best - normalized by the leader's own rate so it reads as a percentage. If the leader dominates, the max is almost always the leader itself and the numerator is ~0. If two variants are tied, the numerator is also ~0 - whenever the other variant wins the draw, it wins by a hair. Regret is only large in the genuinely undecided case: a real gap might exist, and committing now might mean leaving it on the table.
The regret is computed from the same posterior draw that produced the weights - same counts, same fixed seed, same 100,000 draws. The published number never disagrees with the published allocation, by construction. There is no "the weights say one thing but the verdict says another" support ticket waiting to happen; drift between the two is impossible rather than unlikely.
The stopping rule is then one comparison and one sub-case:
- DECIDED when $\text{regret}_{\text{rel}} < 1\%$ - committing to the leader costs less than 1% of its reward rate, in expectation. Then:
- WINNER if the leader's weight (its P(best)) is ≥ 95% - one variant genuinely dominates;
- EQUIVALENT otherwise - the contenders perform the same; P(best) will split forever, but committing costs ~nothing. Pick by taste, brand, or coin flip.
- DECIDING - everything else: still worth waiting.
The EQUIVALENT verdict is the emotional payoff of the whole series. The thrashing of Part 2 and the never-ending test we opened with are the same phenomenon - a ranking question posed to tied variants - finally answered by refusing the ranking frame entirely: these are the same; stop waiting for a winner that doesn't exist.
Two tests - a genuine winner and a truly-equivalent pair - each seen twice, once through the confidence lens and once through the regret lens, make the difference concrete:
The winner test crosses both bars: its leading variant's P(best) climbs past 95% (panel 1) and its regret collapses under 1% (panel 2). The equivalent test crosses only one. Its two variants just swap the lead forever (panel 3) - neither ever reaches 95%, so a confidence rule would keep the test "deciding" indefinitely - but its regret still settles under 1% (panel 4), because with the two rates genuinely tied, committing to either costs almost nothing. Regret closes the test that confidence can't.
Before shipping the thresholds, we calibrated them against every live test in production - a couple dozen at the time. The reassuring finding: the epsilon sits on a plateau. Halving it (1% → 0.5%) completed exactly the same set of tests; every clear winner had regret ≈ 0 and weight ≥ 95% at any reasonable threshold. When your rule's output is insensitive to the exact knob value, the knob is honest - a threshold that only works at one magic value is usually overfit to the data you tuned it on.
Two Sides, One Contract
A small architectural decision here carried a lot of weight. The optimizer only reports the raw number - rel_regret, persisted with each weights version. The thresholds (1% epsilon, 95% winner bar) live on the read side, as constants in the API that serves the test-monitor page. Retuning a threshold is a config-grade change to the reader; the optimizer - the component that owns production traffic - never needs a deploy for it.
The read side mirrors just enough of the optimizer's logic to avoid lying - notably that a stored weight during cold start is a uniform override, not a P(best), so the confidence display stays suppressed until the gates clear.
On top of the verdicts, the monitor derives each test's phase - a lifecycle computed from the data, not stored anywhere: not running (no traffic yet, or gone quiet), learning (inside the cold-start gates, still uniform), running (the optimizer balancing weights), completed (every slot decided), and locked (every slot promoted, ready to convert to static).
The monitor also carries a set of attention flags, designed around a simple principle: a listing page must tell you where to look, not just display state. ready marks a decided slot nobody has acted on. no data marks a test that's been silent for a week (usually a targeting or volume problem). And too few conversions is the interesting one: a pace projection, not a state - at the current click rate, will the slowest cold slot clear its 50-click gate within 30 days? One subtlety - the pace window is min(30, test age) days, so a three-day-old test is judged by its three days of pace, not by a 30-day denominator it hasn't lived yet. Young tests that are ramping up fine shouldn't get flagged for being young.
Closing a Test: Promote-to-Winner
Once a slot reads WINNER (or EQUIVALENT and the user just picks), there's a button: promote. The design principle behind it: promotion is an attribute, not a rewrite. Promoting doesn't convert the template, doesn't rewrite the action, doesn't stop the data. It stamps promoted_variant_id on that slot of the action, and everyone honors the stamp in their own way:
- Send time short-circuits a promoted slot straight from the action payload - the winner holds even if the stored weights are stale or missing entirely.
- The optimizer skips Thompson for that slot and emits a fixed 100/0; the other slots keep balancing normally.
- The weights table gets a
source=promotedversion recording who promoted what, when - the audit trail from Part 1 doing its job.
Because this is one of the few user-driven writes in a machine-owned system, the edge cases got careful treatment. Promoting a different variant into an already-promoted slot returns a conflict error: changing a decided winner must be an explicit reset, never a silent overwrite. But re-promoting the same variant is idempotent, so a client that timed out and retried doesn't get punished for it.
Graduation: From Experiment to Template
The last lifecycle step closes the loop. A test where every slot has been decided and promoted doesn't need to be a test anymore - it's just an email that happens to carry five bandits' worth of machinery. So the final operation converts it: the top-weight variant of each slot (a promoted slot reads 100/0, so its winner is naturally the argmax) gets baked into a brand-new static template with a plain send action; the old dynamic action is locked and redirects to its successor.
There's something quietly satisfying about the endpoint of this whole pipeline being... an ordinary, boring email template. The experiment infrastructure exists to retire itself, one decided slot at a time.
What the Series Taught Us
Three posts, one feature, roughly four months from a hardcoded prototype to a self-retiring experimentation loop. The learnings I'd pack for the next system:
- Decompose before you optimize. The per-slot independence assumption (Part 1) bought a ~125× reduction in data requirements and made everything downstream - rewards, verdicts, promotion - per-slot too. The structure of the decision space was the single highest-leverage choice in the project.
- Separate what's frozen from what's alive. Human-authored content and machine-written weights have opposite lifecycles, so they live apart - the loop owns the weights and never fights the config for the same field.
- Fail open, loudly. The optimization layer can degrade to uniform; it can never block a send. And the kill-switch's compute-only mode means an algorithm change proves itself on live data before it touches an inbox.
- Ask what question your statistic answers. P(best) is for allocating, expected regret is for stopping (Parts 2 and 3). Most of our trouble came from asking one number to do the other one's job.
- Every term must earn its keep. The unsub penalty, the guardrails, the sharpness knob - all felt prudent, all failed measurement, all got deleted. The shipped system is simpler than the first draft.
- Make destruction deliberate. Changing a decided winner is its own endpoint with explicit semantics - a conflict error rather than a silent overwrite, and idempotent on retry.
- Users need verdicts, not posteriors. The most valuable output of all this Bayesian machinery is a word - WINNER, EQUIVALENT, DECIDING - and a flag telling them where to look next.
In the Multi-Armed Bandits post I closed by saying that simple tools, thoughtfully combined, still win. After a year of running bandits in production, I'd sharpen it: the tools were never the hard part. The hard part is deciding what you're actually asking them - and having the discipline to delete everything that doesn't help answer it.