AI agent skill · by @damusix

Arm Bandits Expert

Implements, evaluates, and deploys multi-armed bandit algorithms — including Thompson Sampling, UCB, epsilon-greedy, LinUCB, EXP3, and contextual bandits. Covers algorithm selection, experiment harnesses, offline evaluation (IPS, Doubly Robust), infrastructure patterns, and correctness verification. Use when the user asks about multi-armed bandits, exploration-exploitation tradeoffs, adaptive experiments, A/B testing alternatives, online optimization, bandit-based recommendation or personalization systems, or contextual bandits.

Archived 2026-10-23 · Source updated 2026-08-08. Check the original repository for the latest version.

SKILL.md instructions

# Multi-Armed Bandits Expert

Guide the user based on their actual need. Do not lecture; respond to what they're doing.

## Routing

Assess the user's situation and route to the appropriate reference material. Read the relevant file(s) from `skill/references/` before responding.

### Entry Paths

**"I need to learn about bandits"** → Start with `tier-1-core-algorithms.md`. Progress to tier 2/3 only when the user is ready or asks.

**"I need to pick an algorithm"** → Use the decision framework below, then read the relevant tier reference for details.

**"I need to build/evaluate an experiment"** → Read `experiment-harness-patterns.md` for environment, policy, runner, and offline evaluation abstractions.

**"I need to review/debug an implementation"** → Read `infrastructure-patterns.md` for testing patterns and bug checklists. Cross-reference the relevant algorithm tier for formula verification.

**"I need to deploy to production"** → Read `infrastructure-patterns.md` for serving, reward pipelines, monitoring, and safety guardrails.

**"I need to understand the business case"** → Read `business-applications.md` for domain-specific guidance, real-world examples, and ROI evidence from 60+ named company deployments.

## Decision Framework — Picking an Algorithm

Present trade-offs. Never prescribe a single "best" algorithm without context.

### Step 0: Does the user specify an algorithm?

If the task names a specific algorithm (e.g. "implement UCB1", "use epsilon-greedy"), implement that algorithm — do not substitute a different one. Only use this decision framework when the user asks for help *choosing* an algorithm or says something generic like "implement a bandit."

### Step 1: What kind of rewards?

| Reward type | Candidates |
|---|---|
| Binary (click/no-click) | UCB1 is the classical default; Thompson Sampling (Beta-Bernoulli) for best empirical performance; epsilon-greedy for simplicity |
| Continuous (revenue, time) | UCB1, Thompson Sampling (Gaussian/NIG), LinUCB |
| Adversarial / non-stationary | EXP3, SW-UCB, D-UCB, change-point detectors |

### Step 2: Do you have context features?

| Context | Candidates |
|---|---|
| No context | Epsilon-greedy, UCB1, Thompson Sampling, Softmax |
| User/item features available | LinUCB, contextual Thompson Sampling |
| High-dimensional features | Neural bandits (NeuralUCB/NeuralTS, last-layer Bayesian) |

### Step 3: What's your constraint?

| Constraint | Recommendation |
|---|---|
| Simplest possible baseline | Epsilon-greedy with decay |
| Strongest theoretical guarantees | UCB1 (stochastic), EXP3 (adversarial) |
| Best empirical performance | Thompson Sampling |
| Delayed feedback | Thompson Sampling (robust to stale posteriors) |
| Multiple items per round | Combinatorial bandits (CUCB + oracle) |
| Ranked lists with position bias | Cascading bandits (CascadeUCB1, CascadeLinTS) |
| Arms change state over time | Restless bandits (Whittle index) |
| Reward distributions shift | Non-stationary bandits (SW-UCB, GLR-UCB) |

### Step 4: Maturity reality check

| Algorithm | Maturity | Production examples |
|---|---|---|
| Epsilon-greedy | Battle-tested | Optimizely, Kameleoon |
| UCB1 | Battle-tested | Widespread |
| Thompson Sampling | Battle-tested | Yahoo, Stitch Fix, Doordash |
| LinUCB | Production-proven | Yahoo News, Netflix, Spotify |
| EXP3 | Well-established | Adversarial settings |
| Bayesian UCB | Production-proven | River, MABWiser |
| Softmax | Battle-tested | Deep RL action selection |
| Neural bandits | Early production | Meta ENR (9%+ CTR lift) |
| Non-stationary (SW/D/GLR-UCB) | Well-established | SMPyBandits, monitoring |
| Combinatorial bandits | Research | Ad placement (Chen et al.) |
| Restless bandits | Research → applied | Health interventions (Armman) |
| Cascading bandits | Early production | Expedia homepage ranking |

## Build Phases

1. **Core Library** — Implement algorithms starting with tier 1 (epsilon-greedy or Thompson Sampling). See `tier-1-core-algorithms.md` through `tier-3-production-algorithms.md`.
2. **Experiment Harness** — Build environment, runner, and metrics to compare algorithms offline. See `experiment-harness-patterns.md`.
3. **Production Infrastructure** — Add reward pipelines, serving, monitoring, safety guardrails. See `infrastructure-patterns.md`.

## Reference Files

| File | Contents |
|---|---|
| `references/tier-1-core-algorithms.md` | Epsilon-greedy, UCB1, Thompson Sampling — pseudocode, properties, pitfalls |
| `references/tier-2-practical-algorithms.md` | LinUCB, EXP3, Bayesian UCB, Softmax — context handling, adversarial robustness |
| `references/tier-3-production-algorithms.md` | Neural, combinatorial, non-stationary, restless, cascading bandits |
| `references/experiment-harness-patterns.md` | Environment, policy, runner abstractions, metrics, offline evaluation |
| `references/infrastructure-patterns.md` | Project structure, testing, reward pipelines, serving, monitoring, safety |
| `references/business-applications.md` | Business decision framework, domain guides, algorithm-to-problem mapping, ROI evidence, failure modes |