Monitoring answers questions you thought of in advance: is the service up, is the queue growing, is the disk full. Observability is the property of a system that lets you answer questions you did not think of in advance — why is this particular customer's request slow, only on Tuesdays, only in one region. You need both, and most teams over-invest in the first while assuming they have the second.
The three signals, and what each is actually good at
- Metrics — cheap, aggregated numbers over time. Perfect for "is something wrong" and for alerting. Useless for "which request", because the detail was thrown away at aggregation.
- Logs — expensive, high-detail events. Perfect for the specifics of one occurrence, especially when structured as JSON with consistent field names. Terrible as a substitute for metrics.
- Traces — the path of one request across services, with timing at every hop. The only signal that answers "where did the 800ms go" in a distributed system, and the one teams most often skip.
Instrument with OpenTelemetry. It is the vendor-neutral standard for all three, it has auto-instrumentation for most frameworks, and it means the decision about where the data goes is a config change rather than a rewrite. Propagate trace context across every hop — including through your message queues, which is where traces usually break.
Measure what users experience
CPU and memory are diagnostics, not health. The four signals that describe whether your service is doing its job are latency, traffic, errors, and saturation — and of those, the first three are what a user would notice. Track latency as a distribution, never as an average: a mean of 200ms is entirely consistent with 5% of users waiting four seconds. Look at p50 for the typical case, p95 and p99 for the tail, and remember that in a system where one page makes twenty backend calls, the p99 of each call is the typical experience of the page.
SLI the measurement: proportion of checkout requests served < 500ms
SLO the target: 99.5% over a rolling 28 days
Error budget: 0.5% of 28 days ≈ 3h 22m of failure allowed
Budget remaining → ship features, take risks.
Budget exhausted → freeze risky changes, spend the sprint on reliability.The error budget is the part that changes behaviour. It turns "how reliable should we be" from an argument into a number, and it gives a team a principled way to say both "this is good enough, ship" and "we cannot take that risk this week".
Alerts humans should be woken for
Alert on symptoms, not causes: users cannot see high CPU, they see failed checkouts. Every page must be urgent, actionable, and about something a human can fix right now — anything else belongs on a dashboard or in a ticket. Alert on a burn rate rather than an instantaneous threshold, so that a brief blip does not page anyone but a sustained degradation does, quickly. And treat alert fatigue as an incident in its own right: a rotation that receives thirty pages a week has already stopped reading them, and the one that mattered will be missed.
The practical failure modes
- Cardinality explosions — putting a user ID, request ID, or full URL in a metric label creates a separate time series per value and can take down your metrics backend. High-cardinality detail belongs in traces and logs.
- Logging everything at info level, then paying more for the log pipeline than for the service. Sample aggressively, keep errors at 100%.
- Dashboards nobody reads, built for a launch and never revisited. Keep one service dashboard that answers "is it healthy and where does it hurt" in ten seconds.
- No correlation IDs. Without a shared request ID propagated everywhere, you are grepping across five services by timestamp.
- Monitoring only from inside. Synthetic checks from outside your network catch DNS, TLS, and CDN failures that every internal probe reports as fine.
The test of an observability stack is not what your dashboards look like. It is how long it takes a tired engineer, at 3am, to go from "something is wrong" to "this is the failing component".
Optimize for that number. Everything else — the tooling choice, the dashboard aesthetics, the retention policy — is downstream of it.