Serving

Getting predictions to users inside a budget: batch versus online inference and the hybrid most systems use, the request path stage by stage, latency budgets that add up, approximate nearest-neighbour indexes and their memory, caching, quantisation and distillation, CPUs versus GPUs, a cost-per-prediction model you can defend and the fallbacks that keep a page rendering when a model is slow.