Design ChatGPT
This guide walks through designing a conversational AI system like ChatGPT. We'll follow the Interview Framework to structure our approach.
Phase 1: Requirements
Functional Requirements
- Send messages: Users can send text prompts and receive AI-generated responses
- Conversation history: Users can maintain context across multiple messages in a conversation
- Streaming responses: Users see responses generated token-by-token in real-time
- Conversation management: Users can create, view, and delete conversations
Non-Functional Requirements
- Scale: 150M DAU, 10 requests/user/day = 1.5B requests/day (~17K QPS)
- Latency: Time-to-first-token (TTFT) < 500ms, streaming at 30-50 tokens/second
- Availability: 99.9% uptime (conversational AI is latency-sensitive but not mission-critical)
- Consistency: Eventual consistency is acceptable for cross-device history; read-after-write for the active conversation
Capacity Estimation
Users: 150M DAU
Requests: 150M × 10 = 1.5B requests/day
QPS: 1.5B / 86,400 ≈ 17,000 requests/second
Peak QPS: ~50K requests/second (3x average)
Storage per request:
- Prompt: ~2 KB (~500 tokens × ~4 chars/token)
- Response: ~2 KB (~500 tokens average output)
- Total: ~4 KB per request
Daily storage: 1.5B × 4 KB = 6 TB/day
Annual storage: ~2 PB/year (conversation history)
GPU inference (rough estimate):
- Assume ~100 tokens/second per GPU for a 70B model (conservative baseline)
- With batching + optimizations, real systems achieve 500-1000+ tokens/sec/GPU
- At 17K QPS with ~500 output tokens + ~500 input tokens = ~17M tokens/second
- GPUs needed: 17M / 500 = ~34,000 GPUs (order of magnitude estimate)
- Throughput is dominated by total tokens and context length; this is illustrative
In an interview, GPU estimation is often asked. The key insight is that LLM inference is GPU-bound, not CPU-bound. Mention that you'd batch requests and use techniques like continuous batching to improve throughput.
Phase 2: Data Model
Core Entities
-- Users table
User {
id: UUID (PK)
email: String
created_at: Timestamp
subscription_tier: Enum (free, plus, enterprise)
}
-- Conversations table
Conversation {
id: UUID (PK)
user_id: UUID (FK)
title: String
created_at: Timestamp
updated_at: Timestamp
}
-- Messages table
Message {
id: UUID (PK)
conversation_id: UUID (FK)
role: Enum (user, assistant, system)
content: Text
token_count: Integer
created_at: Timestamp
}
Conversation Context in Redis
Key: conv:{conversation_id}:context
Value: {
messages: [
{ role: "user", content: "..." },
{ role: "assistant", content: "..." }
],
token_count: 4096,
last_accessed: timestamp
}
TTL: 30 minutes
Conversation context is hot data accessed on every request. Store it in Redis for sub-millisecond retrieval. Persist writes asynchronously (Kafka → workers → Postgres) and read from Postgres on cache miss.
Phase 3: API Design
Protocol Choice: HTTP + Server-Sent Events (SSE)
- HTTP POST for sending messages (request-response)
- SSE for streaming responses (server pushes tokens as they're generated)
- WebSocket adds complexity but enables bidirectional features (cancel/interrupt, tool calls); SSE is enough for one-way streaming
Key Endpoints
We support two modes:
- Web app (stateful): Server maintains conversation history; client sends only the new message
- API (stateless): Client sends full message history each request (OpenAI-style)
POST /v1/chat/completions
Request (stateful web app):
{
"conversation_id": "uuid",
"messages": [
{ "role": "user", "content": "Explain quantum computing" }
],
"model": "gpt-4",
"stream": true,
"max_tokens": 4096
}
Request (stateless API):
{
"messages": [
{ "role": "system", "content": "You are a helpful assistant." },
{ "role": "user", "content": "Explain quantum computing" }
],
"model": "gpt-4",
"stream": true,
"max_tokens": 4096
}
Response (SSE stream):
data: {"id":"chatcmpl-123","choices":[{"delta":{"content":"Quantum"}}]}
data: {"id":"chatcmpl-123","choices":[{"delta":{"content":" computing"}}]}
data: [DONE]
---
GET /v1/conversations
Response:
{
"conversations": [
{ "id": "uuid", "title": "Quantum Computing", "updated_at": "..." }
]
}
---
GET /v1/conversations/{id}/messages
Response:
{
"messages": [
{ "role": "user", "content": "...", "created_at": "..." },
{ "role": "assistant", "content": "...", "created_at": "..." }
]
}
---
DELETE /v1/conversations/{id}
Response: 204 No Content
Phase 4: High-Level Design
Architecture Diagram
Request Flow: Sending a Message
- Client sends POST request with user message
- API Gateway authenticates, rate limits, and routes request
- App Server retrieves conversation context from Redis (or Postgres if cache miss)
- App Server forwards prompt + context to Inference Router
- Inference Router selects an available GPU server (least-loaded)
- Model Server generates tokens, streams back via SSE
- App Server forwards streamed tokens to client
- App Server asynchronously persists message to Postgres via Kafka
Component Deep Dive
Inference Router
The inference router is the brain of request distribution:
class InferenceRouter:
def route_request(self, request):
# 1. Select model based on user tier and request
model = self.select_model(request)
# - Free tier → smaller/faster model (e.g., GPT-3.5)
# - Plus tier → can request GPT-4
# - Could also auto-route simple queries to smaller models
# 2. Find available servers for that model
available_servers = self.get_healthy_servers(model)
# 3. Select server based on:
# - Current queue depth (least-loaded)
# - GPU memory utilization
# - Geographic proximity
selected = min(available_servers, key=lambda s: s.queue_depth)
# 4. Apply continuous batching
# Group requests to maximize GPU utilization
return selected.enqueue(request)
Interview follow-up: How would you route requests to different model sizes? You could auto-detect query complexity (simple factual questions → smaller model, complex reasoning → larger model) or let users explicitly choose. This is a cost-latency trade-off.
Model Server (GPU Inference)
Each model server runs:
- Model weights loaded in GPU memory (e.g., 140GB for a 70B model in FP16)
- KV Cache for attention states during generation
- Continuous batching to process multiple requests simultaneously
70B Model Memory Requirements:
- Model weights: 70B params × 2 bytes (FP16) = 140GB
- Requires 4× A100 80GB GPUs with tensor parallelism (or quantization)
Per-GPU Memory (with tensor parallelism across 4 GPUs):
┌─────────────────────────────────────────┐
│ Model Weights (sharded) │ ~35GB
├─────────────────────────────────────────┤
│ KV Cache (shared across batch) │ ~10-30GB
├─────────────────────────────────────────┤
│ Activations + overhead │ ~5-10GB
└─────────────────────────────────────────┘
KV Cache is the bottleneck: Each token in the context requires storing key-value states. With 128K context windows, a single request can consume 10GB+ of GPU memory just for the KV cache.
Tensor parallelism vs Pipeline parallelism: Tensor parallelism splits individual layers across GPUs (lower latency, requires fast interconnect like NVLink). Pipeline parallelism splits layers sequentially across GPUs (higher latency, works with slower interconnects). Most production systems use tensor parallelism within a node and pipeline parallelism across nodes.
Streaming with SSE
async def stream_response(request, model_server):
# Open SSE connection to client
async with sse_response() as response:
# Stream tokens as they're generated
async for token in model_server.generate(request):
await response.send({
"id": request.id,
"choices": [{"delta": {"content": token}}]
})
await response.send("[DONE]")
Phase 5: Scaling & Deep Dives
Deep Dive: GPU Inference Optimization
The biggest challenge in LLM systems is inference throughput. Key techniques:
1. Continuous Batching
Instead of waiting for a batch to complete, add new requests dynamically:
Traditional Batching:
Request 1: [████████████████████]
Request 2: [████████████████████] ← waits for request 1
Request 3: [████████████████████] ← waits for request 2
Continuous Batching:
Request 1: [████████████████████]
Request 2: [██████████████████████] ← starts when slot available
Request 3: [██████████████████] ← starts immediately
2. PagedAttention (vLLM)
The problem: During generation, the model stores key-value (KV) states for every token in the context. Traditional systems pre-allocate a contiguous memory block for the maximum sequence length, wasting GPU memory when actual sequences are shorter.
The solution: Borrow the idea from how operating systems manage RAM—allocate memory in fixed-size pages (blocks) on demand:
- Divide GPU memory into small fixed-size blocks (e.g., 16 tokens each)
- Allocate blocks only as needed as the sequence grows
- Keep a block table mapping each request to its blocks
- Share blocks for common prefixes (e.g., system prompts)
Without PagedAttention: With PagedAttention:
┌──────────────────────┐ ┌──────────────────────┐
│ Request 1 KV Cache │ │ Block Pool │
│ [████░░░░░░░░░░░░░░] │ │ [█][█][█][░][░][░] │
│ (reserved for max) │ │ ↑ ↑ ↑ │
├──────────────────────┤ │ R1 R2 R3 │
│ Request 2 KV Cache │ │ (allocate on demand) │
│ [██░░░░░░░░░░░░░░░░] │ └──────────────────────┘
│ (reserved for max) │
└──────────────────────┘
Result: 2-4x more concurrent requests fit in the same GPU memory.
3. Speculative Decoding
Use a smaller "draft" model to predict multiple tokens, then verify with the main model:
Draft model (7B): predicts tokens [A, B, C, D]
Main model (70B): verifies in parallel
- Accepts [A, B, C] ✓
- Rejects [D] ✗, generates correct token
Result: 4 tokens verified in 1 main model forward pass
Speedup: 2-3x for appropriate draft models
Deep Dive: Context Window Management
With 128K token context windows, we need strategies for long conversations:
Sliding Window + Summarization
def prepare_context(conversation, max_tokens=128000):
messages = conversation.messages
total_tokens = sum(m.token_count for m in messages)
if total_tokens <= max_tokens:
return messages
# Keep system prompt and recent messages
system = messages[0] if messages[0].role == "system" else None
recent = get_recent_messages(messages, max_tokens * 0.7)
# Summarize older messages
old_messages = get_older_messages(messages, recent)
summary = summarize(old_messages) # Call smaller model
context = []
if system:
context.append(system)
context.append({"role": "system", "content": summary})
return context + recent
Deep Dive: Fixed-Window Rate Limits (RPM/TPM)
class FixedWindowLimiter:
def __init__(self, tier):
self.limits = {
"free": {"rpm": 3, "tpm": 40000},
"plus": {"rpm": 60, "tpm": 100000},
"enterprise": {"rpm": 600, "tpm": 1000000}
}[tier]
def check_limit(self, user_id, tokens_requested):
# Check requests per minute
rpm_key = f"ratelimit:{user_id}:rpm"
rpm_count = redis.incr(rpm_key)
if rpm_count == 1:
redis.expire(rpm_key, 60)
if rpm_count > self.limits["rpm"]:
raise RateLimitExceeded("RPM limit")
# Check tokens per minute
tpm_key = f"ratelimit:{user_id}:tpm"
current_tpm = redis.incrby(tpm_key, tokens_requested)
if current_tpm == tokens_requested:
redis.expire(tpm_key, 60)
if current_tpm > self.limits["tpm"]:
raise RateLimitExceeded("TPM limit")
Deep Dive: Content Moderation
- Input moderation: Classify prompts for harmful intent (fast, lightweight model)
- Output moderation: Check generated content before sending to user
- Streaming challenge: Must buffer enough tokens to detect harmful content
Handling Failures
| Failure | Detection | Recovery |
|---|---|---|
| GPU server crash | Health checks every 5s | Inference router retries on different server |
| Long generation timeout | 60s timeout | Return partial response + error message |
| Redis cache miss | N/A | Fall back to Postgres, repopulate cache |
| Kafka lag | Consumer lag monitoring | Scale up workers, apply backpressure |
Trade-offs Discussion
| Decision | Trade-off |
|---|---|
| SSE vs WebSocket | SSE is simpler and sufficient for one-way streaming; WebSocket enables bidirectional features (cancel/interrupt, tool calls) at higher complexity |
| Async persistence | Lower latency but risk of data loss; mitigate with Kafka durability |
| Context in Redis vs every request | Memory cost vs latency; Redis is 100x faster than Postgres |
| Continuous batching | Higher throughput but variable latency per request |
| Smaller draft model for speculation | Uses extra GPU memory but improves end-to-end latency |
Common Pitfalls
Ignoring GPU constraints: LLM inference is fundamentally different from traditional web services. You can't just "add more servers"—GPUs are expensive and memory-limited. Always discuss batching, KV cache, and GPU utilization.
Forgetting streaming: ChatGPT-like systems must stream responses. Waiting for full generation before responding would mean 30+ second delays. SSE or WebSocket streaming is essential.
Underestimating context management: With 128K token contexts, a single request can use 10GB+ of GPU memory. Discuss sliding windows, summarization, or RAG for long conversations.
Treating moderation as optional: Content moderation is critical for production LLM systems. It must happen on both input and output, and streaming makes it tricky.
Synchronous persistence: Writing every message to the database synchronously would destroy latency. Use async writes via a message queue.
Interview Checklist
- Clarified functional requirements (chat, streaming, history)
- Discussed scale (DAU, QPS, token throughput)
- Addressed GPU inference as the main bottleneck
- Explained streaming with SSE
- Covered context management for long conversations
- Discussed batching techniques (continuous batching, PagedAttention)
- Included content moderation
- Addressed rate limiting (RPM, TPM)
- Handled failure scenarios
Summary
| Aspect | Solution |
|---|---|
| Streaming | Server-Sent Events (SSE) for token-by-token delivery |
| Inference routing | Least-loaded GPU selection with continuous batching |
| Context storage | Redis for hot context, Postgres for persistence |
| GPU optimization | PagedAttention, speculative decoding, tensor parallelism |
| Rate limiting | Fixed-window per user tier (RPM + TPM) |
| Persistence | Async writes via Kafka for low latency |
| Moderation | Input + output filtering, buffered streaming |