Design a Rate Limiter for AI Services

Rate limiting for AI inference services (ChatGPT, Claude, Grok, Gemini) presents unique challenges beyond traditional API rate limiting. Users pay for different tiers with vastly different quotas—a free user might get 10 messages per day while an enterprise customer gets millions of tokens per minute. Requests have variable costs (a simple greeting vs. a 100K context analysis), and long-running streaming responses complicate traditional request counting.

This walkthrough follows the Interview Framework. We'll design a rate limiting system that handles tiered quotas, token-based metering, and the distributed challenges of serving millions of users across multiple regions.