Wide events, metrics and logs: picking a shape for telemetry

Every system that runs unattended eventually needs to answer a question nobody anticipated. Why did this customer's run take nine minutes? Which of these failures share a root cause? Did the change we shipped on Tuesday make anything worse?

Whether you can answer depends almost entirely on the shape you chose for your telemetry, and most teams choose it by accident — by installing whatever the framework ships with and adding to it under pressure.

There are three shapes. They are not interchangeable.

Metrics

A metric is a number over time, sliced by a small set of labels. workflow_runs_total{status="failed", kind="webhook"}.

Metrics are cheap in a way nothing else is. Aggregation happens at write time, storage is proportional to the number of distinct label combinations rather than to traffic, and a year of history costs about what a day of raw events costs. They are the correct substrate for dashboards and alerting: is the failure rate above 2% is a metric question, and answering it from raw data is wasteful.

Their limitation is structural and absolute. Once you've aggregated, the individual is gone. A metric can tell you 3% of runs failed. It can never tell you which runs, or what they had in common, because the thing that would let you find out — the identity of each event — is exactly what aggregation discarded.

Worse, the instinct to fix this by adding labels is the fastest way to destroy a metrics system. Add customer_id and you multiply your series count by the number of customers. This is the cardinality problem, and it's why metrics backends stay cheap only if you keep them coarse.

Logs

A log line is a timestamped string, occasionally with structure. Logs are the default because they're free to produce — one function call anywhere in the codebase.

That freedom is the problem. A typical request emits a dozen unrelated lines from a dozen call sites, each with whatever context that call site happened to have. Reassembling one operation means grepping by request ID, if someone remembered to include one. Analysis means regex over unstructured text. The volume is enormous, and the signal density is low: most lines are read exactly never, and the one you need is the one that wasn't logged.

Structured logging improves this considerably — JSON objects instead of strings, queryable fields instead of regex. But it doesn't fix the fragmentation. Twenty structured lines about one operation are still twenty rows you have to join.

Wide events

A wide event is one record per unit of work, carrying everything known about it. Not twenty narrow lines — one very wide row, emitted when the operation completes.

For a workflow run, that single record might carry the run and workflow IDs, the account, the trigger kind, the step count, total duration, duration of the slowest step, which connections were touched, tokens consumed, model used, retry count, final status, error class, deploy SHA, and region.

The properties that follow are the reason the shape is worth the trouble:

Arbitrary questions after the fact. Every field is a dimension you can filter, group and correlate by. P95 duration by model, for accounts on the new plan, excluding runs that retried is a query you can write without having predicted it. With metrics you'd have needed that combination as a series in advance.

Correlation is free. The fields are already on the same row. Noticing that slow runs cluster on one connection kind requires a GROUP BY, not a join across three systems.

High cardinality is fine. customer_id on a wide event is a column value, not a new time series. Columnar stores compress it well. The thing metrics forbid is the thing wide events are for.

One write path. Context accumulates in memory during the operation and is flushed once. Fewer calls, less I/O, less to misconfigure.

The costs are real. Raw event volume is much larger than metrics, so retention is shorter and sampling becomes necessary above some rate. You need a backend that queries columnar data well — a relational database with an index on a timestamp will not do. And it requires discipline at the boundaries: something has to own the per-operation context and guarantee the flush, including on the error path. An event that isn't emitted when the operation crashes is missing precisely when you need it.

What most teams should actually do

Not a choice between three — a deliberate combination:

Wide events as the source of truth for anything you'll want to investigate. One per meaningful unit: a request, a job, a workflow run, an outbound API call.
Metrics derived from those events for dashboards, SLOs and alerts, where cheap long retention and fast aggregation matter more than detail.
Logs for the narrow remainder — process lifecycle, panics, the debug output that genuinely doesn't belong to any unit of work.

The ordering matters. Teams that start with logs and bolt on metrics end up with two systems that disagree and neither of which answers novel questions. Teams that start with wide events can derive the other two.

The test

Take a real incident from the last quarter. Write down the three questions you asked while debugging it. Then check whether your current telemetry could answer them today, without shipping a new log line and waiting for it to recur.