High load is not a technology, it is a set of constraints that break assumptions you did not know you were making. The engineers who are good at it are rarely the ones who know the most frameworks — they are the ones who can estimate, who understand where queues form, and who design for the failure rather than for the happy path.
1. Estimate before you build
Back-of-the-envelope calculation is the most underrated skill in the list. 100 million requests per day is about 1,150 per second average and perhaps 5,000 at peak. A row of 1KB times a billion rows is a terabyte. A memory access is ~100ns, an SSD read ~100µs, a cross-region round trip ~100ms — six orders of magnitude between the first and the last, and knowing that instantly tells you where a design will fall over. Do this arithmetic before choosing anything; half of all architecture debates evaporate when someone produces a number.
2. Understand queueing, at least intuitively
Latency does not degrade linearly with load — it degrades hyperbolically as utilization approaches capacity. At 50% utilization a system feels fine; at 90% queues form and p99 explodes; at 100% it is unbounded. This is why you provision for peak with headroom, why a system that is "only at 85% CPU" is already in danger, and why Little's Law (concurrency = arrival rate × latency) is the most useful formula you will ever memorize. It also explains why adding a retry to an overloaded system makes it worse, not better.
3. Know where the state lives
Stateless services scale by adding copies; state is where scaling gets hard. That means database skills are load skills: indexes and query plans, connection pooling (an unbounded pool is a self-inflicted outage), read replicas and the replication lag that makes read-your-own-writes fail, partitioning and the choice of shard key you cannot easily change later, and transaction isolation levels — knowing what a phantom read is matters the day two workers process the same row.
- Cache-aside, write-through, write-behind — and the eviction and TTL policy for each.
- Stampede protection: request coalescing, jittered expiry, and a lock around the recompute. A popular key expiring at scale is a self-DDoS.
- Cache invalidation as a design decision made up front, not a TODO. It really is one of the hard problems.
- Idempotency keys on every write path that a client can retry — the alternative is duplicate payments.
4. Design for failure, not around it
At scale, something is always broken. The skill is bounding the damage: timeouts on every call (a request without a deadline is a resource leak), retries with exponential backoff and jitter and a strict budget, circuit breakers that stop hammering a dependency that is already down, bulkheads that keep one slow dependency from consuming every thread, and graceful degradation — serving a stale cache or a reduced feature set rather than an error page. Load shedding deserves special mention: rejecting 5% of traffic quickly is dramatically better than serving 100% of it too slowly to be useful.
Every retry is a small DDoS you built yourself. Budget them, cap them, and never retry a request that has already timed out at the caller.
5. Asynchrony and the consistency you can live with
The most reliable way to survive a traffic spike is to not do the work synchronously. Queues and event streams turn a spike into a backlog, which is a problem you can wait out. That brings its own competencies: at-least-once delivery means consumers must be idempotent, ordering guarantees are per-partition, consumer lag is a first-class metric, and dead-letter queues need an owner. It also brings eventual consistency, and the real skill is negotiating which parts of the product can tolerate it — a like counter can be stale, a balance check cannot.
6. Measure, then optimize
- 01Profile before optimizing. Intuition about hot paths is wrong more often than it is right.
- 02Load test with realistic data volumes and realistic distributions — a uniform key distribution hides every hotspot you will actually have.
- 03Track percentiles, never averages, and track them per endpoint.
- 04Run a game day: kill a node, add 300ms of latency to a dependency, fill a disk. You will find assumptions no design review would have caught.
- 05Write down the capacity model and revisit it quarterly, so growth is a planned event rather than a surprise.
The last skill is judgment: knowing when not to do any of this. Most systems never reach the load that justifies sharding, and premature distribution buys you all of the complexity with none of the benefit. Build the simplest thing that can be measured, measure it, and let the numbers decide when the next step is due.