Design38 min
Capping how many requests a client can make per time window.Pushing updates to clients the moment they happen.Keeping hot data in fast storage to skip slow lookups.Splitting data across machines so no single node holds it all.A language model in a loop that calls tools and acts on the results.
PremiumChatGPT
A billion people a week type a message and watch the answer stream back, and every one of those answers occupies a GPU for its whole length.
What they’re really testing: Every reply holds GPU memory for as long as it streams, and GPUs are the one resource you can't add by the weekend. How do you keep the first token under a second for a million streams in flight, and degrade the free tier before any paid reply stutters?
Premium
This design is on the Premium plan
Readers with Lifetime Access read it in full. Already bought it? Sign in from the top bar.
What’s inside
- Primary concepts and the hard part
- Requirements
- API / Model
- High-level architecture
- Trade-offs and deep dives
- Possible follow-up questions