Drowning in Logs: Why Collecting More Data Is Making Your Team Blind
There's a particular kind of suffering that happens in an on-call rotation when your monitoring stack is technically world-class and practically useless. You open your log aggregator. There are millions of lines. The error is in there somewhere. Probably. And now it's 2:47am and you're scrolling through a wall of JSON like you're looking for a specific grain of sand on a beach.
This is the observability paradox: the more data you collect, the harder it gets to see anything clearly.
It sounds counterintuitive. Shouldn't more information mean better visibility? In theory, yes. In practice, your logging pipeline has become a firehose pointed directly at your engineers' faces, and the signal you actually need is buried under gigabytes of noise you're paying to store and never reading.
The Signal-to-Noise Trap
Most teams arrive at observability overload the same way. Something breaks in production. The post-mortem surfaces a gap — you didn't have enough visibility into X. So you add more logging around X. Then Y breaks and you do the same. Over time, you've instrumented everything, everywhere, all at once.
The problem is that logging decisions made reactively during incident reviews tend to optimize for "we should have had this" rather than "we actually need this going forward." You end up with a system that logs every database query, every HTTP header, every cache miss, and every retry — not because you have a coherent observability strategy, but because past-you was scared and present-you is paying for it.
The result? A signal-to-noise ratio that's basically noise. Engineers learn to distrust the monitoring stack because finding a useful log line requires either superhuman grep skills or dumb luck. Alerts fire constantly because you've set thresholds on metrics that were never meaningful to begin with. And somewhere, a very expensive Elasticsearch cluster is humming along storing data nobody reads.
Log Cardinality: The Problem That Sneaks Up on You
Cardinality is one of those words that sounds like infrastructure jargon until it personally costs you $40,000 in observability tooling bills.
High-cardinality data means you're logging values with massive variance — user IDs, session tokens, request GUIDs, IP addresses — as label dimensions or index fields. Every unique value creates a new time series or index entry. Do that across enough services and enough request volume and you've turned your metrics platform into a combinatorial explosion that's slow to query, expensive to store, and structurally hostile to the kind of "show me what's different about these failing requests" investigation you actually need to run.
The fix isn't to stop capturing high-cardinality data entirely. It's to be intentional about where it lives. High-cardinality fields belong in your trace spans and structured log payloads — not as metric label dimensions. Your dashboards should aggregate. Your traces should detail. Conflating the two is where things go sideways.
The Lean Observability Mindset
Teams that actually debug fast don't necessarily have more data. They have better-shaped data.
Lean observability starts with a simple question: what decisions does this data need to support? If you can't answer that, you probably shouldn't be emitting it. Every log line, every metric, every trace should exist because it helps an engineer answer a specific class of question faster. "We might need this someday" is how you end up with a 400TB retention bill and zero clarity.
Here's a practical framework that works:
Define your debug narrative first. Before you instrument anything, write down the questions you'd need to answer during an incident. What failed? Where in the call chain? For which users or tenants? Since when? Those questions should map directly to the data you're capturing. If you're logging things that don't connect to any of those questions, cut them.
Separate your observability layers. Metrics are for alerting and trending. Logs are for context. Traces are for causality. These are not interchangeable. Teams that dump everything into logs and call it observability are doing a lot of grep-based archaeology when they should be doing structured root cause analysis.
Sample aggressively, trace selectively. You do not need a full trace for every successful 200 response. Sample your happy path at 1-5%. Trace 100% of errors, slow requests, and anything touching a critical path. This alone can cut your observability costs by half while actually improving your ability to debug, because your trace data is now dense with signal instead of diluted with noise.
Set log levels like you mean it. DEBUG logs should not be running in production by default. Ever. If your app is emitting debug-level output in prod because someone forgot to change an environment variable two years ago, that's not observability — that's a liability. Audit your log levels quarterly. It takes an hour and saves you from drowning.
The Alert Fatigue Feedback Loop
Here's something nobody talks about enough: bad observability doesn't just slow down debugging — it actively degrades your team's ability to respond to real problems.
When alerts fire too often, engineers stop treating them as urgent. When dashboards are cluttered, engineers stop checking them proactively. When log searches take three minutes and return 50,000 results, engineers start solving problems by instinct and tribal knowledge instead of data. You've built a monitoring stack that's technically running but functionally abandoned.
Alert fatigue is the behavioral consequence of bad signal-to-noise ratio. And it's dangerous because it's invisible until something serious gets missed. The team isn't ignoring alerts because they're lazy — they're ignoring them because the alerts have trained them to. That's an infrastructure problem disguised as a culture problem.
The fix is ruthless prioritization. Audit every alert currently firing more than twice a week. If it's not actionable — meaning an engineer can look at it and know exactly what to do — it either needs to be rewritten or deleted. An alert that requires context, interpretation, and a Slack thread to understand is not an alert. It's a riddle.
Building Toward Clarity
The goal of observability isn't to capture everything. It's to make the right things findable fast.
That means treating your monitoring stack like a product with real users — your engineers — and designing it around their actual workflows. What does an on-call engineer need in the first five minutes of an incident? Start there. Build backward from that. Cut everything that doesn't serve it.
The teams shipping fast and sleeping well aren't running the biggest observability pipelines. They're running the most intentional ones. Fewer metrics, higher fidelity. Fewer dashboards, better questions. Less data, more clarity.
Your logs don't need to know everything. They just need to tell you what matters when it matters. That's the whole job.