Rate Limiting

How a distributed rate limiter works, step by step

1. A request arrives at the gateway or service middleware. The limiter derives a key: API key for authenticated traffic, IP address for anonymous traffic, plus the route, for example rl:key_abc:/search.
2. It looks up the rule for that key, such as 10 requests per second with a burst of 20.
3. It performs an atomic check-and-update in Redis using a Lua script, so two gateway instances cannot both spend the last token.
4. If allowed, the request proceeds and the response includes headers like X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset.
5. If denied, the limiter returns 429 Too Many Requests with a Retry-After header, and logs the event for abuse monitoring.
6. If Redis itself is unreachable, the limiter follows a configured policy: usually fail open (allow traffic) for public APIs, or fall back to a coarse local limit.

Algorithms compared

class TokenBucket {
    private final double capacity;      // max burst, e.g. 20
    private final double refillPerSec;  // steady rate, e.g. 10
    private double tokens;
    private long lastRefillNanos;

    TokenBucket(double capacity, double refillPerSec) {
        this.capacity = capacity;
        this.refillPerSec = refillPerSec;
        this.tokens = capacity;
        this.lastRefillNanos = System.nanoTime();
    }

    synchronized boolean tryAcquire() {
        long now = System.nanoTime();
        double elapsedSec = (now - lastRefillNanos) / 1e9;
        tokens = Math.min(capacity, tokens + elapsedSec * refillPerSec);
        lastRefillNanos = now;
        if (tokens >= 1) {
            tokens -= 1;
            return true;
        }
        return false;
    }
}

The distributed version stores tokens and lastRefill in a Redis hash and runs the same arithmetic inside a Lua script, which Redis executes atomically. Note that the bucket is refilled lazily on each request rather than by a background timer, so idle keys cost nothing but memory, and a TTL cleans them up.

When to rate limit (and when something else fits better)

Rate limit public APIs, login and OTP endpoints (to slow credential stuffing), expensive operations such as search or report generation, and outbound calls to third parties that enforce their own quotas. Rate limiting is not a substitute for capacity planning or autoscaling, and it is a blunt tool against distributed attacks from millions of IPs, where edge DDoS protection does better. For internal overload protection, load shedding based on server health (reject when queue depth or CPU is too high) often works better than fixed per-client numbers.

Common mistakes in interviews

• Keeping counters in each server's memory behind a load balancer, which multiplies the effective limit by the number of servers.
• Using a non-atomic read-then-write on Redis, creating a race that lets bursts through.
• Not deciding what happens when the limiter's store is down.
• Limiting only by IP, which punishes users behind a shared NAT or corporate proxy and is trivially bypassed by attackers with many IPs.
• Forgetting to tell clients when to retry, which causes aggressive retry loops that make overload worse.

Interview questions