When a team says “we use a bandit for personalisation,” that sentence is usually standing in for a system that doesn’t exist yet. The bandit is the part that feels like the product — it picks the action, the action is what the user sees — so it gets built first and everything else gets bolted around it. Then the system looks fine in offline evaluation, feels mediocre in production, and nobody can quite say why.

The reason is almost always the same. The policy is not the centre of a decision system. The belief is.

The shape of the problem

You are making a sequence of decisions about someone whose true state you cannot see — a learner’s actual understanding, a customer’s real position and intent, a patient’s underlying condition. You never observe the state directly. You infer it from noisy clues: what they answered, how long they took, what they did last week. Your actions change that state. And the outcome you care about — did they pass, did they reach the goal — arrives late, sometimes months late, as a single sparse signal.

That structure has a name: a partially observable Markov decision process, a POMDP. Hidden state, dynamics your actions drive, delayed reward, observations that are clues rather than truth. You don’t need the formalism to build the system. You need to take the “partially observable” part seriously, because it’s the part most architectures quietly drop.

The wrong centre

The instinct is to put the policy in the middle. Context goes in, an action comes out, a reward updates the policy. The belief about the user — what they know, what they need — gets computed somewhere upstream and flattened into a feature vector before it reaches the policy.

That flattening is where the system stops compounding. Once the belief is a vector of point estimates, every downstream consumer is working with a number that has thrown away how sure you were. You can still ship it. It’ll even look fine on a dashboard. But the policy now has no way to tell the difference between “this learner is at the median, and we’re confident” and “this learner is at the median, and we have almost no idea” — and those two situations call for completely different actions.

The right centre

Put the belief in the middle instead. The belief is a posterior: a distribution over the hidden state, a most-likely value together with how uncertain you are. Mean and variance at minimum, the full posterior if you can carry it. Everything else in the system is a consumer of that belief — the policy that picks the next action, the model that projects the state forward to exam day, the component that scores a mock, the generator that makes new content. They all read the same belief. The policy is one consumer among several, and not the most important one.

Every consumer reads the same belief — that shared read is the contract.

This reframe moves two things from the margins to the centre. Estimating the belief — item response theory, knowledge tracing, hierarchical priors that let sparse concepts borrow strength from their parents — becomes first-class architecture, not preprocessing. And validating the decisions — proving the policy caused the outcomes rather than riding a trend — becomes first-class too, not “we’ll run A/B tests at some point.” Both are where the advantage compounds, because any team can wire up a bandit. Almost nobody builds the disciplined belief-estimation and causal-validation loop around it.

The loop, in one pass

The teaching loop — or the guidance loop, or whatever your domain calls it — has seven stages:

1. Observe — turn raw activity into a clean event stream.
2. Estimate — update the belief over hidden state. Output a distribution, not a point.
3. Project — push the belief forward in time, because what matters is recall on exam day, not accuracy today.
4. Decide — given the projected belief and the constraints, pick the next action. This is where the bandit lives.
5. Generate — produce the content that delivers the chosen action.
6. Act — deliver it, and choose when.
7. Validate — did the decision cause the gain, or was it going to happen anyway?

Seven stages, one of them the policy. The belief produced at stage two and projected at stage three is the contract that ties the rest together.

Why the contract has to carry uncertainty

Pass the belief as a distribution between stages. Never as a point estimate. Each downstream stage breaks in its own way when you throw the variance away:

• Projection bends. Forgetting curves are non-linear, so the projected recall of an ability you’re unsure about is not the projection of the average ability — Jensen’s inequality guarantees a systematic bias.
• The policy stops exploring. Thompson sampling and upper-confidence-bound methods both take uncertainty as their input. Hand them a point estimate and they collapse to greedy: exploration disappears, and the policy stops learning.
• Adaptive testing loses its objective. Computerised adaptive testing picks the next item to shrink the variance of the ability estimate. No variance, no objective — it degenerates into picking at random.
• Validation goes blind. The effects you most want to find — a policy that helps the uncertain learners but not the confident ones — are invisible unless per-user uncertainty is in the logs.

Why the variance has to travel with the estimate. The curve maps latent ability to retention. Two beliefs share the same mean, so a point estimate scores them identically (the dot). But the true expected retention — the average of the curve over the belief — sits lower, and drops further as the belief widens. Throw away the variance and every projection inherits this bias.

The failure mode that kills teams is upstream of all of this, and it’s mundane. Someone designs the data schema around point estimates because the mean is what the UI shows. A database column mastery FLOAT can’t carry variance. An API contract { mastery: 0.7 } can’t either. Once that ships, the uncertainty is gone from the whole system and no clever model downstream can recover it. Design the contracts to carry distributions from the first day — mean and variance per concept at minimum, sufficient statistics if you can.

Boundaries between learned and rule-based parts

A belief-centric architecture has a practical payoff: it lets you keep most of the system simple. Build the dumbest end-to-end loop that works — rules at every stage, basic logging — and get something running before anything is learned. Then replace one stage with a learned component only when measurement shows that stage is the bottleneck. Rules at the session level, a small bandit at the item level, calibrated estimation in the middle. A working stack with rules everywhere beats a half-built stack with one fancy policy and nothing around it.

This only holds if the contracts between stages are clean. The belief is the interface. As long as a stage consumes a belief and emits a belief — or an action — you can swap a rule for a model behind that interface without touching anything else. Learned and non-learned components evolve independently because the contract between them stays stable.

Validation is the moat

The last stage is the one most systems never build, and the one that matters most. You can’t run a clean A/B test for every change — randomisation is rarely possible in production, and you don’t want to experiment on people when the stakes are real. Off-policy evaluation is the way out: re-weight the data you already logged to estimate what a new policy would have done, before you deploy it. Inverse propensity scoring is the foundation; doubly-robust estimators, which stay correct if either your propensity model or your outcome model is right, are the sensible default. Log the propensity of every action from day one, or you won’t be able to do any of this later.

Pair that machinery with a posture: audit every win with at least one causal estimator before you believe it. Pre/post comparisons lie. A result that survives a doubly-robust estimate and a sceptical null is one you can build on. A result that only survives a dashboard is not.

What this buys you

A decision system under hidden, evolving state is three things working together: an honest belief about what you know and how sure you are, a decision that respects that uncertainty instead of pretending it away, and a way to check yourself before you ship. The policy — the bandit everyone wants to talk about — is the easy part. The belief and the validation around it are the work, and they’re what compounds.

↑ Back to the top