Debug Like a Scientist, Not a Gambler

There are two ways to chase a bug. The first: change something, rerun, squint at the output, change something else. It feels productive because you're always moving. The second: make a precise claim about what's broken, then design the cheapest possible test that could prove you wrong. The first is gambling. The second is science. Six years in, I've done plenty of both, and the house always wins the first game.

The night the towers "died"

A while back I was building tower-degradation analytics for a telecom operator. The dashboard's job was simple to describe: watch per-tower metrics, compare them against a rolling baseline, raise an alert when something degrades. One morning the ops channel lit up. Hundreds of alerts, seemingly every tower in a region degrading at once, all timestamped shortly after midnight. Either half the network fell over during the night, or our platform was lying.

The network was fine. The platform was lying. And the tempting move at 9 a.m., with stakeholders watching, was to start tweaking: loosen the thresholds, widen the baseline window, add a debounce, redeploy, wait a day. Every one of those is a coin flip. Some would even have "worked" by burying the symptom.

Instead we made ourselves reproduce it first. We pulled the raw metrics for one noisy tower into a local PostgreSQL instance and replayed the aggregation job against it. The flood came back, deterministically, every time. That single step changed everything: a reproducible bug is a fact you can interrogate.

A bug you can't reproduce isn't a bug yet. It's a rumor.

With a reproduction in hand, the pattern jumped out. The alerts didn't cluster at midnight local time. They clustered at 00:00 UTC, which is 4 a.m. in Muscat. That's not noise. That's a signature.

One hypothesis, the cheapest experiment

We wrote the claim down in one sentence: the aggregation job buckets metrics by UTC day, while the baseline is computed in local time, so for four hours after 00:00 UTC the "today" bucket is nearly empty and every tower looks like it dropped 90%.

Notice what that sentence does. It's specific enough to be wrong. If we reran the replay with both sides forced to the same timezone and the flood persisted, the hypothesis was dead, and we'd have learned something either way.

The experiment cost ten minutes: one SET TIME ZONE 'UTC' on the replay database session, rerun, watch. The flood vanished. The actual bug was a single expression, a DATE(recorded_at) in a SQL view that silently used the session timezone, feeding a Python job that did everything else in UTC. Two components, each internally consistent, disagreeing about when "today" starts.

The fix was boring. Finding it wasn't luck, it was procedure:

• Reproduce first. Deterministically, on the smallest slice of data that still shows the behavior.
• State one falsifiable hypothesis. If you can't say what evidence would kill it, it's a hunch, not a hypothesis.
• Run the cheapest experiment that can falsify it. Not the most thorough one. The cheapest.
• Bisect the space. Every experiment should eliminate roughly half the remaining possibilities: which pipeline stage, which time window, which commit. git bisect is this idea with a UI.

Gambling explores one configuration per attempt. Bisection kills half the search space per attempt. Over a long debugging session that's not a difference in style, it's exponential.

Logs are your lab notebook

None of this works if the system can't testify about its own behavior. The only reason we spotted the 00:00 UTC signature is that our alerts carried full timezone-aware timestamps instead of naive local times. That was discipline someone had paid for months earlier.

The habits I no longer compromise on: log timestamps with offsets, always. Log inputs at system boundaries, because that's where two components' assumptions meet, and that's where they disagree. And log decisions, not just events: "alert raised because baseline=X, current=Y, threshold=Z" turns a mystery into a query.

Reading code you didn't write

The broken view was written by someone who had left the company, and the first instinct in that situation is to distrust the code and judge the author. Resist both. Read it the way a scientist reads field notes: the code is the ground truth of what the system actually does. Not the docs, not the diagram in Confluence, not your memory of a meeting. The person who wrote DATE(recorded_at) wasn't careless; they tested it on a server that happened to run in local time, and it was correct there. The bug wasn't in the line. It was in the seam between two reasonable assumptions.

That's the real argument for debugging like a scientist. Gamblers need luck, and luck doesn't transfer. A hypothesis log transfers. Six months later, when a "duplicate rows around midnight" ticket showed up in a completely different pipeline, the first question anyone asked was: whose midnight?