Monitoring & Alerting
SLOs, error budgets, and alerts that fire on user pain rather than CPU graphs.
35 minDifficulty 3/5ops · reliabilityAI-writtenWritten by a model on 16 August 2026 and not yet read by a person. Checked automatically: schema, the pedagogical rules the hand-written material is held to, and every diagram parsed for real.
Before this
Why this exists
An on-call engineer gets paged at 3am because CPU usage crossed 80% — a number that, on its own, says nothing about whether any actual user was affected. Meanwhile, a genuine outage where checkout silently fails for every user produces no alert at all, because nobody defined a check for it. Alerting on infrastructure metrics that don't map to user pain trains the whole team to ignore pages — which is worse than having no alerts at all.
The mental model
An SLO (Service Level Objective) is a target stated in terms users actually experience — '99.9% of checkout requests succeed within 2 seconds' — not an infrastructure metric like CPU or memory. The ERROR BUDGET is the allowed amount of failure implied by that target (0.1% of requests may fail or be slow) — and alerting exists to warn you BEFORE that budget is exhausted, not to report every infrastructure fluctuation that doesn't actually threaten it.
How it works
Alert on symptoms users feel, not on causes that might not matter yet
High CPU usage MIGHT be a problem, or might just mean the system is efficiently using available capacity under normal load — it's a potential CAUSE, not a guaranteed symptom of user pain. 'Checkout success rate dropped below 99%' or 'p99 latency exceeded 3 seconds' are actual SYMPTOMS users are experiencing right now, and are what alerts should be built around, with causes investigated only once a real symptom fires.
Alert fatigue from noisy, low-value alerts is a genuine, measurable danger
An on-call engineer who gets paged for things that turn out not to matter, repeatedly, starts treating EVERY page with less urgency — including the one that's a genuine, severe outage. Every low-value alert has a real cost: it erodes the team's trust in the alerting system as a whole, which is exactly why alert QUALITY matters more than alert QUANTITY.
The error budget reframes reliability work as a tradeoff, not an absolute
A 99.9% SLO means roughly 43 minutes of acceptable downtime per month, by definition — an error budget still remaining means the team can reasonably ship a riskier change or take a planned maintenance window; a budget already exhausted means the priority shifts to reliability work FIRST, before shipping new, potentially risky features. This turns 'how careful should we be right now' into a concrete number instead of a vague feeling.
A dashboard you check is different from an alert that finds you
A dashboard is useful for investigating a known problem or getting a general sense of system health when you choose to look — it requires someone to actively check it. An alert is what tells you about a problem you WEREN'T looking for, at the moment it starts mattering, which is the specific job dashboards structurally can't do on their own.
The mechanism
An SLO defines an acceptable target stated in user-experienced terms, deriving an error budget — the amount of failure that target implicitly tolerates. Metrics tracking that specific user-facing symptom (success rate, latency) are continuously evaluated against thresholds derived from the SLO, and crossing a threshold that meaningfully threatens the error budget triggers an alert, routed to whoever's on call, ideally with enough context to begin investigating immediately.
What people get wrong
- More alerts, covering more possible failure modes, always means better monitoring.
- Alert VOLUME beyond what's actually actionable produces fatigue, which measurably degrades response quality to genuine incidents — a smaller number of well-chosen, symptom-based alerts that people actually trust and act on beats a large number that gets tuned out. Teams new to monitoring often over-alert defensively ('just in case'), not realizing this actively makes their incident response WORSE by training people to deprioritize pages.
- Infrastructure metrics like CPU and memory usage are the right things to alert on directly.
- These are potential CAUSES of a problem, not guaranteed symptoms that users are actually affected — high CPU with no impact on actual request success or latency doesn't need to wake anyone up; alerting should target the user-experienced symptom, investigating infrastructure metrics as part of DIAGNOSING an already-triggered alert. This is exactly why 'alert on symptoms, not causes' is a foundational monitoring principle — causes like CPU usage are useful for debugging, not for deciding whether to page someone.
- A 100% uptime target is always the right goal to aim for.
- Pursuing ever-higher reliability has real, accelerating costs (more redundancy, more caution, slower feature velocity), and for most systems, a target below 100% (like 99.9%) that reflects actual user needs is a more honest and practical goal than an unachievable, infinitely expensive absolute. The error budget concept exists specifically because 100% is the wrong target for almost everything — it treats every minute of downtime as equally catastrophic, when in reality reliability investment has diminishing returns past a certain point relevant to actual user experience.
When not to use it
- You're investigating a known, already-reported problem and want to understand the system's current state.
- A dashboard — actively checked, giving a broad view of current metrics, appropriate for investigation rather than for being proactively notified of an unknown problem.
- A metric is genuinely useful for post-incident diagnosis but doesn't, on its own, indicate a user-facing problem when it changes.
- Track it on a dashboard for investigative use, without configuring an alert on it directly — reserve alerts for symptoms that actually warrant waking someone up.
Terms
- SLO (Service Level Objective)
- — A target for system reliability or performance stated in terms of actual user experience, like request success rate or latency.
- Error budget
- — The amount of failure an SLO implicitly tolerates, used to balance reliability work against the pace of shipping new features.
- Alert fatigue
- — The degraded response quality that results from too many low-value or false-positive alerts, causing genuine incidents to be treated with less urgency.
- Symptom-based alerting
- — Alerting on metrics that directly reflect user-experienced problems (success rate, latency) rather than potential underlying causes (CPU, memory).
In an interview
Your team wants to add an alert for every metric that COULD indicate a problem, arguing more coverage is always better. How would you push back on this?
- excessive low-value alerts cause alert fatigue, which measurably degrades response to genuine incidents as people learn to deprioritize pages
- recommend alerting on user-facing symptoms (error rate, latency) rather than every possible underlying cause
- causes like CPU or memory usage are better suited to dashboards for investigation once a real symptom-based alert has already fired
Can you recall it?
Why should alerts be built around user-facing symptoms rather than underlying infrastructure causes like CPU usage?