Rate Limiting - Why Your Service Keeps Saying 429

Introduction

Ever hit a 429 Too Many Requests and immediately started complaining about why this is happening to you? Congratulations, that was your not-so-friendly introduction to rate limiters.

Almost every big product used by millions of people runs one, for two main reasons:

1. Cost - you don't want a fleet of servers spun up under your name just because a handful of clients decided to hammer you. Someone is paying that bill.
2. Security - protection from DoS (denial of service) attacks. Intentional or not, one user (or "user") can take up all your capacity by firing hundreds of thousands of requests and overwhelming the servers. Stopping that before it reaches your application is a very good idea.

There's a third reason that gets less attention: fairness. Without limits, one noisy client degrades the experience for everyone else sharing the same infrastructure.

A rate limiter solves all three by putting a cap on requests - per user, per IP, per API key, or per endpoint. Something like a user can only make 2 requests per second if you're posting to Instagram or Facebook.

Where Does the Limiter Live?

Rate limiting can be applied on the client side or the server side. Client-side limits are a nicety - they make well-behaved clients back off gracefully - but they're trivially bypassed, so they're not a defense. In this post we'll stick to the server side.

On the server side it usually sits as middleware in front of your actual handlers, or one layer further out at the edge:

  Client A ──┐
  Client B ──┼──▶  Load balancer  ──▶  Rate limiter
  Client C ──┘                               │
  (DoS)                                      ├── under limit ──▶  Your service
                                             │
                                             └── over limit  ──▶  429 Too Many Requests

The further out you push it, the less work your infrastructure does for requests it was going to reject anyway. A limiter that runs after authentication, database lookups, and business logic has already cost you most of what you were trying to save.

You Might Not Need to Build This

Before writing any code: every cloud provider ships an API Gateway service with rate limiting, IP whitelisting, quotas, and a pile of other features already wired up. Reverse proxies like NGINX and Envoy do it too. For a lot of teams that's the whole answer.

Building your own is a different commitment. Now you own the optimization story, and if you're running a distributed system, you own synchronization as well. Do it when you need limits that off-the-shelf tools can't express - tiered plans, per-tenant quotas, dynamic limits that change based on load.

The Algorithms, Briefly

There are five classics worth knowing:

• Token bucket - tokens drip into a bucket at a fixed rate, each request consumes one, an empty bucket means rejection. Allows short bursts up to the bucket size.
• Leaking bucket - requests queue up and drain at a constant rate. Smooths traffic into a steady stream, but no bursts and requests can sit waiting.
• Fixed window counter - count requests per fixed interval, reset at the boundary. Dead simple and memory-cheap, but a client can fire a full window's worth on either side of the boundary and effectively double the limit.
• Sliding window log - store a timestamp per request and count what falls inside the trailing window. Very accurate, but you're storing every request.
• Sliding window counter - a hybrid that weights the previous window's count against the current one. Nearly as accurate as the log at a fraction of the memory.

Token bucket, drawn out:

        refill: N tokens per second
                     │
                     ▼
            ┌────────────────────┐
            │   bucket - cap C   │
            └────────────────────┘
                     │
  incoming request ──┴──▶  token available?
                                  │
                                  ├── yes ──▶  consume token, forward request
                                  │
                                  └── no  ──▶  reject with 429

Quick comparison:

If you're picking one and moving on, token bucket or sliding window counter will serve you well.

Where Do the Counters Live?

For every request you obviously don't want to go back to your primary database to check how many requests this client has left - that adds latency to every single call, on the hot path, for something you're doing purely to save resources.

In-memory is the natural starting point. It's as fast as it gets and costs you nothing. It also stops working the moment you run more than one pod:

                     ┌──▶  Pod 1   local counter: 2
  Load balancer ─────┼──▶  Pod 2   local counter: 2
                     └──▶  Pod 3   local counter: 2

  limit is 2/sec  ·  client actually got 6/sec

Each pod only sees the slice of traffic that was routed to it, so your effective limit gets multiplied by the number of pods. Worse, it drifts as you autoscale.

The fix is a shared store that is fast and genuinely good at expiring data. That's Redis:

  Pod 1 ──┐
  Pod 2 ──┼──▶  Redis  ──▶  one counter per client, TTL-backed
  Pod 3 ──┘

Sub-millisecond reads, native TTLs so old windows clean themselves up, and atomic operations that let you increment and check in one round trip.

Going Distributed

Once multiple limiters share a counter, the interesting problems show up.

Race conditions. Two pods read count = 9 against a limit of 10, both decide there's room, both write 10. You've served 11 requests. Reaching for a mutex feels natural, but locks are expensive on a path this hot, and a plain in-process mutex does nothing across pods anyway.

The answer is to make the check and the increment atomic inside Redis itself:

• Fixed window - INCR returns the new value, so a single command both increments and tells you whether you're over. Pair it with EXPIRE on first write.
• Sliding window log - a sorted set per client, with timestamps as scores. ZREMRANGEBYSCORE drops everything older than the window, ZCARD counts what's left, ZADD records the new request.
• Anything more involved - put the whole read-decide-write sequence in a Lua script. Redis runs it as a single atomic unit, so no other client observes a half-applied state.

Here's the sliding-window version as a Lua script, which is the shape I usually reach for:

-- KEYS[1] = rate limit key, e.g. "rl:user:1234"
-- ARGV[1] = now (unix millis)
-- ARGV[2] = window size in millis
-- ARGV[3] = max requests in window
local clearBefore = tonumber(ARGV[1]) - tonumber(ARGV[2])

redis.call('ZREMRANGEBYSCORE', KEYS[1], 0, clearBefore)

local count = redis.call('ZCARD', KEYS[1])
if count < tonumber(ARGV[3]) then
    redis.call('ZADD', KEYS[1], ARGV[1], ARGV[1] .. ':' .. math.random())
    redis.call('PEXPIRE', KEYS[1], ARGV[2])
    return 1
end

return 0

And calling it from a Go middleware:

var slidingWindow = redis.NewScript(luaScript)

func RateLimit(rdb *redis.Client, limit int, window time.Duration) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            key := "rl:" + clientID(r)
            now := time.Now().UnixMilli()

            allowed, err := slidingWindow.Run(r.Context(), rdb,
                []string{key}, now, window.Milliseconds(), limit).Int()

            // Fail open: if Redis is down, don't take the whole API down with it.
            if err != nil {
                next.ServeHTTP(w, r)
                return
            }

            if allowed == 0 {
                w.Header().Set("Retry-After", "1")
                http.Error(w, "rate limit exceeded", http.StatusTooManyRequests)
                return
            }

            next.ServeHTTP(w, r)
        })
    }
}

Redis becomes a dependency on your hot path. Every request now does a network round trip, and Redis itself becomes a single point of failure. Two things help: keep Redis in the same region or availability zone as your services so the hop stays sub-millisecond, and decide explicitly whether you fail open or fail closed when it's unreachable. Failing open keeps the API alive during a Redis outage but leaves you unprotected; failing closed protects you but converts a cache outage into a full outage. Most public APIs fail open.

Latency cost. If even one round trip per request is too much, the usual compromise is a local counter per pod that syncs with Redis periodically, accepting some slop in exchange for speed. You give up exactness - which is fine, because rate limits are a blunt instrument anyway. Nobody is harmed if a client occasionally gets 102 requests instead of 100.

Don't Forget the Response

A rate limiter that rejects requests without explaining itself is an unpleasant thing to integrate against. Send back enough for clients to behave:

Retry-After is the important one - without it, clients retry immediately, and your rejection path starts taking as much traffic as your success path did.

Things Worth Getting Right

• Choose the key carefully. Per-IP breaks for users behind corporate NATs or mobile carrier gateways, where thousands of people share an address. Per-user is better once a request is authenticated; per-API-key is better for machine clients.
• Not every endpoint deserves the same limit. A search endpoint that fans out to three services should be cheaper to exhaust than a health check.
• Return 429, not 503. They mean different things, and monitoring tools treat them differently.
• Log the rejections. A spike in 429s is either an attack or a legitimate client that outgrew its tier. Both are worth knowing about.
• Test the boundaries. Most rate limiter bugs live at window edges and in the very first request for a fresh key.

Conclusion

Rate limiting looks trivial until you have more than one pod. A counter in a map is a fifteen-minute job; making that counter correct across a fleet, fast enough to sit on every request, and resilient when its data store disappears is the actual work.

Start with the managed option if it fits. If it doesn't, pick token bucket or sliding window counter, keep the state in Redis, make the check-and-increment atomic with a Lua script, and decide up front what happens when Redis is unreachable. That covers the large majority of what production actually throws at you.