System Design · Medium
Design an api gateway
Route, authenticate, rate-limit, and observe all external traffic to your microservices.
Requirements
What the system must do, and how well it must do it.
Functional
Route requests to backend services by path, header, or query.
Authenticate: JWT validation, API keys, mTLS, OAuth2 introspection.
Rate limit per client, route, or globally (token bucket).
Request/response transformation: rewrite paths, add headers, JSON ↔ protobuf.
TLS termination, certificate management, and SNI routing.
Observability: access logs, metrics (latency, errors), distributed tracing.
Non-functional
p99 added latency under 5 ms (excluding backend).
Handle 100K RPS per gateway instance; horizontal scaling.
Config changes (routes, rate limits) hot-reloaded without restart.
Zero-downtime deploy: draining in-flight requests.
Estimation
Back of the envelope numbers that drive the design.
10K routes, 1M clients. Config size < 10 MB.
Gateway fleet: 50 instances × 100K RPS = 5M RPS capacity.
Rate limit counters in Redis (10M keys, TTL = window).
TLS certs: 10K certs, auto-renewed via ACME (Let's Encrypt).
Architecture
Design decisions
Architecture
The gateway is a stateless fleet (Envoy, Kong, NGINX, or custom Go/Rust). Each request passes through a filter chain: TLS → Auth → Rate limit → Router → Transformation → Upstream. Filters are pluggable; order matters. Config is stored in a control plane (etcd, Consul, or DB) and pushed to gateways via xDS or long-poll.
Authentication
JWT: validate signature (JWKS cached), check exp, aud, iss, and custom claims. API key: lookup in Redis/DB, check scopes. mTLS: verify client cert against CA. OAuth2 introspection: call auth server (cached). Failed auth returns 401 with WWW-Authenticate header. Auth results (user_id, scopes) are passed to upstream via headers.
Rate limiting
Token bucket per (client_id, route) or (client_id, global). Counters in Redis with Lua script for atomic check-and-decrement. Returns 429 with Retry-After, X-RateLimit-Limit, X-RateLimit-Remaining. Config: rate, burst, window. Distributed: each gateway instance shares Redis; local cache for hot keys reduces Redis load.
Routing and transformation
Routes match by path prefix, regex, header, or query. Upstream: service name (resolved via service discovery) or static host. Load balancing: round-robin, least-request, consistent hash (session affinity). Transformation: path rewrite, header add/remove, body transform (JSON ↔ protobuf via gRPC transcoding). Timeout, retry, and circuit breaker per route.
Questions and answers
Follow-ups an interviewer is likely to ask. Try answering before you open each one.
How do you handle a backend service outage?
Circuit breaker: after 5% 5xx errors in 10 sec, open the breaker. Return 503 immediately for a cooldown period (30 sec). Half-open: allow probe requests. Fallback: static response or cached response for GET.
What about WebSocket and gRPC?
HTTP/2 upgrade for gRPC (h2c or TLS). WebSocket upgrade passthrough. The gateway must support long-lived connections and stream framing. Timeouts: idle timeout for streams, per-message timeout for gRPC.
How do you do canary releases?
Route 5% of traffic to the new version (by header, cookie, or random). Mirror: copy request to canary, discard response. Compare error rates and latency. Gradually shift traffic.