Logging & Observability
Structured logs, metrics, traces — being able to answer 'what happened?' at 3am.
40 minDifficulty 3/5ops · debuggingAI-writtenWritten by a model on 16 August 2026 and not yet read by a person. Checked automatically: schema, the pedagogical rules the hand-written material is held to, and every diagram parsed for real.
Before this
Why this exists
A customer reports "checkout failed" at 3am, with no error message, no timestamp beyond "a while ago," and no request id — and the only logs available are `console.log('here')` scattered through the codebase, printed with no context about WHICH request, WHICH user, or WHEN. Observability is the discipline of instrumenting a system so that "what happened?" has an actual, specific answer, before you need it at 3am.
The mental model
Three complementary signals, each answering a different question: LOGS answer 'what specifically happened, in detail, for this one event' — a rich but expensive-to-search record. METRICS answer 'how is the system doing, in aggregate, over time' — cheap, numeric, great for graphs and alerts. TRACES answer 'where did time go, across every service this one request touched' — following a single request's journey through a distributed system.
How it works
Structured logs are searchable; plain text logs are only readable
`console.log('User 5 failed login')` is readable by a human scrolling through output, but nearly useless for querying — 'show me every failed login for user 5 in the last hour' requires actual text parsing. `logger.info('login_failed', { userId: 5, reason: 'bad_password' })` as structured JSON can be queried directly: filter by `event: 'login_failed'` and `userId: 5`, instantly, across millions of log lines.
A correlation id ties every log line from one request together, across services
Assigning a unique id to each incoming request, and including that SAME id in every log line generated while handling it — even across multiple downstream services — lets you filter for exactly that one id and see the request's complete story, in order, regardless of how many separate services or log files it touched.
Metrics are cheap precisely because they discard detail
A counter incrementing 'requests_total' or a histogram tracking 'request_duration_ms' doesn't record any information about a SPECIFIC request — no user id, no request body — just an aggregate number updated in place. This is exactly what makes metrics cheap to store and fast to query even at massive scale, and exactly why they can't answer 'what happened to THIS specific user's request,' which is what logs and traces are for.
A trace shows where time actually went across a distributed request
A request touching an API gateway, an auth service, a database, and a third-party payment API generates one SPAN per hop, all linked under one trace id — visualizing this shows exactly how long each hop took and reveals, say, that 90% of a slow request's total time was spent waiting on the payment API, not the application's own code, which no amount of application-level logging alone would make as immediately obvious.
The mechanism
An incoming request is assigned a unique correlation id, propagated through every function call, log statement, and downstream service call made while handling it. Structured log entries, metrics counters/histograms, and trace spans are all emitted throughout the request's lifecycle, tagged with this id and other relevant context. When something goes wrong, that id lets you pull the complete, ordered story of exactly this one request across every system it touched.
What people get wrong
- Having lots of console.log statements throughout the code counts as good observability.
- Unstructured, untagged log output is hard to search, impossible to correlate across a specific request reliably, and expensive to query at scale — volume of logging isn't the same as USEFUL logging, which requires structure and correlation ids. A codebase riddled with console.log calls often FEELS well-instrumented while actually being nearly useless for answering 'what happened to this specific failed request' during a real incident.
- Metrics alone are sufficient observability, since dashboards show whether the system is healthy.
- Metrics show aggregate trends (error rate spiked at 3pm) but can't answer 'why', or 'which specific requests were affected' — that requires drilling into logs or traces for the actual detail metrics deliberately discard for efficiency. A metrics-only setup can tell you SOMETHING is wrong but leaves you unable to actually diagnose what, which is why logs and traces exist as complementary, not redundant, signals.
- Adding a trace to every single function call provides maximally useful observability.
- Excessive, overly fine-grained tracing adds real overhead (storage, processing cost) and can bury the genuinely useful signal (which hop was slow) under noise from spans that individually took microseconds and tell you nothing interesting. Effective tracing targets meaningful boundaries — service calls, database queries, external API calls — not every internal function, which is a common overcorrection once a team starts adopting tracing.
When not to use it
- You need to know whether the system, in aggregate, is healthy right now, for an at-a-glance dashboard.
- Metrics — they're cheap, fast to query, and exactly designed for this aggregate, real-time health signal, unlike logs which are too detailed and expensive to scan for a live dashboard.
- You need to understand exactly what happened for one specific, reported incident affecting one user.
- Structured logs filtered by that request's correlation id, and a trace if the request spanned multiple services — metrics alone can't reconstruct one specific request's story.
Terms
- Structured logging
- — Emitting log entries as queryable structured data (like JSON with named fields) rather than freeform text, enabling precise filtering and aggregation.
- Correlation id
- — A unique identifier assigned to a request and propagated through every log entry and service call made while handling it, tying them together.
- Metric
- — An aggregate numeric measurement (a counter, gauge, or histogram) tracking system behaviour over time, cheap to store and query but without per-request detail.
- Trace / span
- — A trace follows one request across every service it touches; each span represents one hop or operation within that trace, showing where time was actually spent.
In an interview
A request is slow, and you suspect it's a downstream service call, but application-level logs show the request itself completing quickly on your side. How would you confirm where the time is actually going?
- a distributed trace would show a span for each hop the request makes, including downstream service calls
- comparing span durations reveals which specific hop is consuming most of the total time
- application-level logs alone can't show this, since they only capture what happens on this one service, not the full cross-service picture
Can you recall it?
Why do logs, metrics, and traces exist as three separate, complementary signals rather than one being sufficient on its own?