Infrastructure
Observability — logs, metrics, and traces
When one server did everything, debugging was easy: read the log. Now a single click fans out across a gateway, three services, a database, and a cache — and "it's slow" could be any of them. Observability is how you regain the ability to ask your running system why it did something, after the fact. It rests on three complementary pillars.
The three pillars
- Logs — timestamped records of individual events. "User 42 checkout failed: card declined." Great for the detail of what happened at one point, terrible for spotting trends across millions of them.
- Metrics — numbers aggregated over time: requests per second, error rate, p95 latency, memory used. Cheap to store and perfect for dashboards and alerts — they tell you something is wrong and when, but not why.
- Traces — the path of a single request as it hops across services, broken into timed spans. This is what pins down where the time actually went.
A trace is the "where"
Each span records a service, a start time, and a duration, and nests inside its parent. Laid out as a waterfall, a trace makes the answer obvious: the wide bar is the slow one. Without it, you'd be guessing which of five services to blame; with it, you can see the database query was 80% of the request before you've opened a single log.
Metrics tell you that something's wrong and when. Traces tell you where — which service ate the time. Logs tell you what exactly happened there. You need all three.
How they work together
The three form a debugging funnel. A metric alert fires: p95 latency doubled at 14:20. You pull up a slow trace from that window and see the orders service's database span ballooned. You open the logs for that span and find a query missing an index after a data spike. Metric → trace → log: from "something's wrong" to the exact line, in three steps.
It's the natural complement to the resilience patterns — circuit breakers and proxies keep a struggling system standing; observability tells you which part was struggling and why. Build it in early: the worst time to discover you can't see inside your system is the night it breaks.