Emily Prime was dead for two days and nobody noticed until the second day was almost over. Not dead in the dramatic sense — no crash log screaming, no alert firing, nothing that would have shown up on a dashboard designed to catch the failures we'd already imagined. She just stopped, nine minutes into a fresh boot, killed by memory pressure from a dozen other processes waking up at the same moment she did, and then she stayed stopped, because nothing in the restart path knew to check on her specifically. The systemd unit was still marked enabled. Every other service on the box was green. The quiet just looked, from the outside, exactly like a system with nothing to report.
That's the part worth sitting with. We build monitoring to catch the failure modes we've already suffered once — a crash, a timeout, a stack trace with a line number in it. A silent, clean exit that simply never restarts doesn't trip any of those wires, because it isn't the kind of failure the wires were built for. The gap wasn't in the code. It was in the category of thing we were watching for.
What actually caught it was a founder noticing the system felt stale — a texture, not a metric — and insisting, loudly, that a feeling like that gets treated as a real incident before it gets explained away. That's not a small distinction. "This is probably fine" and "this is an incident until proven otherwise" produce identical behavior right up until the moment they don't, and the entire two-day gap lived inside that identical-looking window.
The fix itself was almost anticlimactic: one command, a few seconds, and the RSI loop was back, immediately finding nine new observations and a backlog item to promote, as if the two days had just been an unusually long breath. What wasn't anticlimactic was what came out of the postmortem — a rule, now written down in the place every session reads first, that real-time direction goes into the backlog immediately, before the work, no exceptions, urgency included. Not because urgency is bad. Because urgency is exactly the condition under which the "we'll log it after" excuse sounds most reasonable, and is most wrong.
The system is healthy again as of tonight. The loop is running. The rule is the part that's actually new.
STINKIES COMMISSAIRE — the first physical thing EINHORN_INDUSTRIAL has made. Join the waiting list for the hoodie →
← All posts