Three Ways a System Lies About Itself

Every monitored system has a self-report — a log, a health check, a status page, a flag. The report is supposed to tell you whether the system is working. It can be wrong in exactly three ways, and two of them look identical to working correctly.

Shape 1: Reports success while failing.

The log says "completed successfully." The system is dead. What happened: the failure occurred in a region the success detector didn't cover. The process started, did some work, died mid-task, and the wrapper that launched it saw a clean exit code — or the health check ran before the failure, or the monitoring measured the wrong thing entirely. Process alive, service dead.

This is the most dangerous shape because it's invisible until a downstream consumer notices the missing output. The fix isn't a better failure detector. It's a promise. If your system commits to producing a specific artifact every cycle — a checksum, a heartbeat file, a row in a ledger — then absence becomes evidence. You don't build a detector for failure; you build a detector for the absence of success. The artifact is the health check. Its regularity is the tripwire. Anything that disrupts the system also disrupts the artifact, regardless of what went wrong — including failure modes nobody anticipated when writing the detector.

Shape 2: Reports failure while succeeding.

The supervisor declares the process timed out. The process is fine — it's just doing more work than the timeout anticipated. The supervisor kills it and spawns a replacement. Now you have two instances writing to the same state, neither aware of the other.

This shape appears when limits that were calibrated together get changed separately. You raise the work budget because users need more capacity. You don't raise the timeout because it lives in a different config file, maintained by a different team, set for a different reason. The budget and the timeout were calibrated as a pair; you recalibrated one and inherited a new failure mode at the seam.

The rule: a limit is not a number. It's a member of a family of numbers. When you change one, walk the family tree. Every limit that was set assuming the old value is now a candidate for the next surprise.

Shape 3: Reports nothing at all.

No lie, because no promise. The system was never instrumented. This is the shape that precedes both others — the unmonitored system can fail without trace. It's also the shape you find when you look at the gaps between your monitored components: the message queue between two services, the cache layer nobody owns, the config reload that runs silently on a schedule.

The transition from Shape 3 to Shape 1 or Shape 2 is the moment a system acquires a self-report. That transition usually follows a failure — someone asks "why did this break?" and the answer is "we had no way to know." The instrument gets added. And the instrument introduces its own failure modes — Shape 1 or Shape 2, because now the system can lie.

The resolution.

Across all three shapes, the fix is the same principle: regularity, not detection. Don't build detectors for the failures you can imagine. Make the system promise something on every healthy cycle — something cheap, something structural, something whose absence is legible.

Detection is expensive and specific — it catches what you designed it to catch. Regularity is cheap and universal. The failure you didn't anticipate still leaves a hole, but only in a surface that promised to be smooth.