Scaling a GenAI Pilot from 50 to 50,000 Users
The pilot works for 50 users and the customer wants 50,000 - what actually changes: caching, model routing, prompt and context budgets, prefix caching, streaming, and cost per request as a first-class metric.
Last updated:
The pilot works beautifully for 50 users and the customer wants 50,000. Add more servers fails this question, because at a thousand times the users the binding constraints are cost per request, provider rate limits and tail latency - and all three are dominated by tokens: how many you send, how often you avoid sending them at all, and which model receives them. The interviewer is checking whether you reach for infrastructure or for the token budget.
“Add more servers fails this one. At 1000x the users, cost per request and tail latency become the design. Instrument tokens, cost, cache hit rate and time to first token before optimising anything, because every lever below is unmeasurable without them.”
Clarifying Questions (Ask These First ~5 min)
| Question | Why it matters |
|---|---|
| 50,000 registered, or 50,000 concurrent? | Two or three orders of magnitude sit between those numbers |
| What is the request mix and the peak-to-average ratio? | Provisioning is set by peak; cost is set by average |
| What is the budget per user per month? | Turns make it cheaper into an actual target |
| What latency does the user perceive? | Time to first token, not total completion time, when the UI streams |
| How repetitive are the requests? | Cache hit rate is the single biggest cost lever available |
| What provider rate limits apply - tokens and requests per minute? | Rate limits, not compute, are usually the first wall you hit |
| Is any quality regression acceptable in exchange for cost? | Routing to a smaller model is a quality decision, and someone must own it |
Architecture (Draw This)
Say this as you draw it
Key Components (30-second pitch each)
- 1Cost per request is a first-class metric - Record tokens in, tokens out, model, cache hit and computed cost on every request from day one, and publish it per feature. Without it every optimisation argument is a guess, and product decisions get made without a price tag attached.
- 2Two cache layers with different risk profiles - Exact-match on a normalised prompt is free of quality risk and catches genuinely repeated questions. Semantic caching catches paraphrases, carries real risk, and must be tenant-scoped, time-bounded and never applied to account-specific or time-sensitive answers.
- 3Model routing with the big model on the hard tail - Classify by task and input length, send the bulk to a small fast model, and escalate on low confidence, validator failure or explicit complexity signals. Routing usually saves more than any prompt tweak, and the escalation path is what protects quality while it does.
- 4Prompt and context budgets - The system prompt is sent on every request, so each wasted token is multiplied by all traffic - trim it once and it pays forever. Cap retrieved context by tokens rather than document count, and truncate conversation history deliberately rather than when the provider complains.
- 5Prefix caching - Put everything static - system prompt, tool definitions, few-shot examples - at the front and keep it byte-identical so the provider can cache it. Cached input tokens are dramatically cheaper and cut time to first token, and a single stray timestamp in the prefix silently disables the whole benefit.
- 6Streaming changes what you optimise - Streaming does not make the system faster, it makes it feel faster. Once the UI streams, time to first token is the user-facing number and total completion time becomes a cost metric, so track and alert on them separately.
What Separates a Strong Answer
- 1Prioritize token budget - A strong candidate focuses on managing token usage effectively, as it directly impacts cost, provider limits, and latency. Infrastructure scaling is secondary to optimizing token flow.
- 2Implement dual caching - Using both exact and semantic caches optimizes response times and cost. Exact caching is risk-free, while semantic caching requires careful management to avoid degrading answer quality.
- 3Utilize model routing - Routing requests based on complexity ensures that resources are used efficiently. The small model handles easy tasks, while the large model is reserved for complex queries, balancing cost and quality.
- 4Monitor and alert costs - Real-time cost monitoring and alerts on cost per request help prevent surprise bills. This proactive approach allows for adjustments before costs escalate, maintaining budget control.
What Breaks First As You Scale
| Wall | Symptom | Fix |
|---|---|---|
| Provider rate limits | 429s at peak while your own CPUs sit idle | Multiple keys and regions, queueing with backpressure, per-tenant quotas |
| Cost per request | Unit economics invert somewhere around 10x users | Caching, routing, prompt trimming, prefix caching |
| Tail latency | p50 looks fine, p99 is terrible | Timeouts and hedged requests, a smaller model for long inputs, streaming |
| Retrieval layer | Vector search p95 climbs as the corpus grows | Filter before search, shard, cache embeddings |
| Stateful sessions | Conversation state pinned to one node | Externalise session state to Redis |
| The human layer | Review queues and support load scale linearly with users | Automate triage or plan the headcount honestly |
Savings Levers, Ranked
| Lever | Typical effect | Quality risk |
|---|---|---|
| Exact-match cache | A meaningful share of requests never reach a model | None |
| Small-model routing | The largest single cost reduction on routed traffic | Managed by the escalation path |
| Prefix caching | Large discount on repeated input tokens plus better TTFT | None |
| Prompt and context trimming | 10-40% of input tokens | Low, if evaluated |
| Semantic cache | A further modest share of requests | Real - wrong answers to similar-but-different questions |
| Batch tier for offline work | Roughly half price at most providers | Latency only - never for interactive paths |
3 Biggest Risks
- 1The semantic cache quietly degrading answers - A threshold that looked safe in testing returns yesterday's answer to a subtly different question. Mitigated by an explicit false-hit budget measured on labelled near-duplicate pairs, a short TTL, and a hard exclusion for anything account-specific.
- 2Routing regressions nobody measures - Cost falls, quality falls with it, and nobody notices for a month. Mitigated by gating every routing change on the evaluation suite, shadow-running before switching, and keeping a per-route quality dashboard beside the per-route cost dashboard.
- 3The surprise bill - Spend is discovered monthly, on an invoice. Mitigated by per-tenant quotas, real-time cost attribution, and alerts on cost per request rather than only on total spend - the ratio moves before the total does.
Google Stack: Cloud Load Balancing + Cloud Armor rate limiting → Memorystore for the exact cache → Vertex AI with context caching → Gemini Flash for the routed bulk and Gemini Pro on the tail → Cloud Monitoring custom metrics for cost per request → BigQuery for token and cost attribution
Azure Stack: API Management (quotas and per-tenant throttling) → Azure Cache for Redis → Azure OpenAI with provisioned throughput units for the predictable base and pay-as-you-go for the spill → a small deployment for routed traffic and GPT-4o on the tail → Application Insights + Cost Management
AWS Stack: API Gateway with usage plans → ElastiCache → Bedrock with prompt caching and cross-region inference profiles → Claude Haiku for the routed bulk and a larger Claude on the tail → Bedrock batch inference for offline work → CloudWatch metrics and Cost Explorer with per-tenant tagging
Other Options: LiteLLM or an in-house gateway for routing, retries, key rotation and per-tenant budgets across providers | GPTCache or a Redis vector index for semantic caching | vLLM on your own GPUs once steady volume on a small model makes the break-even calculation favour it.
Frequently asked questions
Do you need GPUs or self-hosting at that scale?
Do the arithmetic out loud rather than expressing a preference. At 50,000 users making a few requests a day against a small model, hosted APIs are usually still cheaper than idle GPUs and avoid the operational burden. Self-hosting wins when volume is high, steady and served by a small model, or when residency forces it.
How do you set the semantic cache threshold?
Empirically, on a labelled set of near-duplicate and confusable pairs, against an explicit false-hit budget agreed with the business. Then keep an escape hatch: never semantically cache anything account-specific, time-sensitive or personalised, regardless of similarity score.
How do you roll out the router without a quality regression?
Shadow mode first - route the request, serve the large model's answer, and compare offline. Then canary by percentage with the evaluation suite gating promotion, and keep the ability to pin any route back to the large model instantly.
Should you fine-tune a small model to replace the large one?
That is the end state of routing rather than the starting point. Collect large-model outputs on your own routed traffic, fine-tune, and hold the result to the same evaluation bar. Budget for the retraining treadmill, because each base-model update restarts the work.
What happens during a provider outage?
A second provider behind the same interface, prompts kept to the portable subset, and a degraded mode that says so plainly. Test the failover on a schedule, because prompts are not actually portable until you have run them somewhere else.
Where does the first engineering week go?
Instrumentation: tokens, cost, cache hit rate and TTFT per route and per tenant. Every other lever on this page is unmeasurable without it, and teams that skip it spend the next month arguing about which optimisation to try first.