System design deep dive
Caching: deep dive
Caching keeps hot data in memory so most reads never touch the database, cutting latency from milliseconds to microseconds and shrinking database load by an order of magnitude. Getting it right is mostly about invalidation, stampedes, and knowing what you can afford to serve stale.
How a cache-aside read and write work, step by step
- A request for product
42arrives at the app server, which builds a cache key such asproduct:42:v3(the version suffix lets you invalidate an entire schema change at once). - The app issues
GET product:42:v3to Redis. A hit returns in roughly 0.2 to 0.5 ms over the network, and the request is done. - On a miss, the app queries the database (typically 2 to 20 ms), serializes the row, and writes it back with
SET product:42:v3 <json> EX 600so it expires after 10 minutes. - When the product is updated, the app first commits the change to the database and then deletes the cache key rather than overwriting it.
- The next read misses, reloads fresh data, and repopulates the cache. Deleting instead of setting avoids a race where two concurrent writers leave an older value in the cache.
The arithmetic is what sells caching in an interview. With a 95% hit rate, only 1 in 20 reads reaches the database, so a database that handles 5,000 reads per second can now back roughly 100,000 reads per second at the application tier.
Where caches live
- Client and browser: HTTP
Cache-Controlheaders, free but hard to invalidate. - Edge: a CDN for static assets and cacheable API responses.
- In-process: a local map or Caffeine/Guava cache inside each server. Fastest possible, but every instance has its own copy, so invalidation is per instance.
- Distributed: Redis or Memcached shared by all app servers. One source of truth for cached values, at the cost of a network hop.
- Database internal: the buffer pool and query caches the database already maintains.
When to use it (and when not to)
Cache data that is read far more often than it is written and that tolerates a little staleness: user profiles, product catalogs, rendered pages, session data, computed leaderboards. Be cautious with data where a stale read causes real harm, such as account balances or inventory counts at checkout; read those from the source of truth. Also skip caching when the access pattern has no locality, for example a long tail where almost every key is requested once, because the hit rate will be too low to justify the memory.
Classic cache failure modes
| Problem | What happens | Mitigation |
|---|---|---|
| Cache stampede | A hot key expires and thousands of requests hit the DB at once | Single-flight lock per key, or refresh early before expiry |
| Cache penetration | Requests for keys that do not exist always miss and hit the DB | Cache negative results briefly, or front with a Bloom filter |
| Cache avalanche | Many keys share the same TTL and expire together | Add random jitter to TTLs (for example 600s plus up to 60s) |
| Hot key | One key gets so much traffic it overloads a single cache node | Replicate the key, add a local in-process layer for it |
| Stale after write | A race leaves old data cached after an update | Delete on write, short TTL as a safety net, versioned keys |
Product getProduct(long id) {
String key = "product:" + id;
Product cached = redis.get(key);
if (cached != null) return cached;
// Only one caller per key rebuilds; others wait briefly and retry.
if (redis.setIfAbsent("lock:" + key, "1", Duration.ofSeconds(5))) {
try {
Product fresh = db.findProduct(id);
int ttl = 600 + random.nextInt(60); // jitter avoids avalanches
redis.set(key, fresh, Duration.ofSeconds(ttl));
return fresh;
} finally {
redis.delete("lock:" + key);
}
}
sleep(50);
return getProduct(id); // retry, usually a hit now
}Common mistakes in interviews
- Adding a cache box without saying what is cached, the key format, the TTL, or the expected hit rate.
- Never discussing invalidation, which is the hard part interviewers want to hear about.
- Using write-behind for data that cannot be lost, without acknowledging the durability risk if the cache node crashes.
- Assuming the cache always holds everything. Size it against the working set: caching the top 20% of 10 million objects at 2 KB each needs about 4 GB.
- Ignoring what happens when the cache cluster is down. The database must survive a cold-cache restart, or you need gradual warm-up.
Interview questions
›Why delete the cache entry on update instead of writing the new value?
Two concurrent updates can finish their database writes in one order and their cache writes in the other, leaving the older value cached indefinitely. Deleting forces the next reader to load from the database, which is always current. A short TTL bounds the damage if a delete is ever lost.
›How do you choose between LRU and LFU eviction?
LRU evicts what has not been touched recently and adapts quickly to shifting popularity, which suits most web workloads. LFU keeps items that are consistently popular even if they were not used in the last few seconds, which protects against a one-off scan flushing the cache. Redis supports both via maxmemory-policy.
›How would you keep in-process caches consistent across 50 servers?
Keep TTLs short, and broadcast invalidation events through pub/sub (Redis pub/sub or a message queue) so each server evicts its local copy. For data that must be consistent everywhere, skip the local layer and use only the shared distributed cache.
›What is the difference between Redis and Memcached for caching?
Memcached is a simple multi-threaded key-value store with very predictable performance. Redis adds rich data structures (sorted sets, hashes, streams), persistence, replication, and Lua scripting, which makes it useful for leaderboards, rate limiters, and queues as well as plain caching.
›How do you warm a cache after a cold start?
Preload the most popular keys from access logs or a precomputed list before taking traffic, and shift traffic onto the new cluster gradually. Request coalescing ensures that the misses that remain do not all reach the database at once.