Observability on a budget: five costly mistakes

Most observability bills don't grow because the tools are expensive. They grow because nobody decided what to keep, so teams keep everything, alert on everything, and pay for dashboards that nobody opens during an incident. If you run a 10–50 person engineering team behind a B2B product, this is usually one of the top four items on your cloud invoice and the one that gets the least scrutiny.

Mistake 1: Keeping every log line forever

Set a level and retention policy before you pick a log tool, because raw volume is the largest and most fixable cost in a typical stack. Turn off DEBUG in production, drop health-check and readiness-probe lines at the shipper, and route only errors and warnings to hot storage. Everything else belongs in cheap object storage with a 30–90 day life.

A worked example: a 15-service platform writing 20 GB of logs a day. Roughly two-thirds of that comes from load balancer health checks and an ORM logging every query. Mute those two sources and you are down to 7 GB a day — 210 GB a month instead of 600 GB. Even at a modest per-GB rate, that is a two-thirds cut on your biggest line item without losing a single error.

One caution: logs routinely hold phone numbers, emails and order IDs. Under the DPDP Act, a 90-day hoard of raw personal data is a liability, so mask fields at ingest rather than trying to clean up later.

Mistake 2: High-cardinality labels that explode series count

Labels should describe the shape of the system, not the people using it — the moment a user ID, order ID or raw URL becomes a label, your series count multiplies and so does the bill. Every unique combination of label values is a separate time series, and managed tools charge per active series.

Keep production metrics to five or six labels: service, environment, region, endpoint group, status class, and maybe deploy version. Put user and order identifiers in traces or logs, where they belong and where you can actually query them by ID.

The numbers escalate quickly. One request-latency metric carrying a user_id label across 40,000 monthly active users is not one series — it is 40,000 per endpoint, per status class. That single decision can move a metrics bill by an order of magnitude, and almost nobody queries it.

Mistake 3: Uniform trace sampling

Trace sampling should be biased rather than uniform: keep 100% of errors and slow requests, and a small random slice of the rest. Head sampling at 10% is quick to set up and almost always wrong, because the one checkout that took 14 seconds is exactly the trace you sampled away.

Use tail sampling in the OpenTelemetry Collector instead. The collector buffers spans for a short window, then keeps every trace with an error flag or a duration above your threshold and discards most of the healthy ones. You end up storing roughly 1–5% of spans and still hold the evidence for every incident.

Watch the span multiplier when you budget. A single checkout request can emit 50–80 spans across your services, so two million requests a month is well over 100 million spans before any sampling at all.

Mistake 4: Paging people for things that cannot be fixed at 2am

An alert should fire only when a user is being harmed and someone can act on it within ten minutes. CPU at 80% is not that. Error rate climbing on the payments endpoint is. Alert on symptoms your customers feel, not on causes your infrastructure happens to report.

Sort your existing alerts into three buckets and be ruthless:

• Page: error rate over threshold for 5 minutes, p95 latency past the SLO, payment success rate dropping.
• Ticket: disk at 80%, certificate expiring in 14 days, queue depth trending up for an hour.
• Delete: sustained CPU, single-pod memory, anything without an owner or a runbook link.

In our experience a team of five to eight engineers can absorb two to four pages a week per on-call engineer. Past that, people start swiping notifications away, and the one real outage gets missed.

Mistake 5: Buying a platform before writing down the questions

Write down the five questions you need answered at 2am, then buy the cheapest tool that answers them. For most B2B products those questions are: is it up, is it slow, which deploy broke it, which customer is affected, and what is this costing us. That list rarely justifies an enterprise contract.

A self-hosted Grafana, Prometheus and Loki stack on a modest cloud VM covers uptime, metrics and logs for a surprising number of teams; we have written before about what that looks like in practice on your own servers. The step most teams skip is mapping how data actually flows between services before choosing tools — an hour spent on system architecture saves weeks of instrumenting the wrong things. Managed platforms earn their price once you cross roughly 30 services or run a real on-call rota, and cloud spend should be reviewed separately from tool choice, because egress and storage often cost more than the licence.

Where to start

Pick one service and give it a week: turn off DEBUG logging, mute health checks at the shipper, set 14-day hot and 90-day cold retention, cap metric labels at six, and switch on tail sampling. Measure the bill before and after — most teams find the first 40–60% of the cut is free. If you want a second pair of eyes before signing a contract, Maysan Labs does this kind of instrumentation work as part of building and running platforms, and we are happy to compare notes.