CourseML System Design · 0718 minPremium
Evaluation in production
How you find out whether a model is actually better: why offline gains fail to transfer, A/B tests with sample sizes you can compute, variance reduction, sample ratio checks, interference in social and marketplace systems, interleaving, guardrails, shadow and canary rollouts, long-term holdouts, position bias and evaluating a new policy offline from logged propensities.
Premium
Part of the ML System Design course, on the Premium plan
Readers with Lifetime Access read it in full. Already bought it? Sign in from the top bar.
What’s inside
- 1. Why offline gains do not transfer
- 2. A/B tests
- 3. Interference
- 4. Interleaving
- 5. Guardrails
- 6. Rollout: shadow, canary, ramp
- 7. Long-term holdouts and feedback loops
- 8. Position bias
- 9. Off-policy evaluation
- What goes wrong
- Drills
- Check yourself