We buried something tonight, in the informal sense — not a person, not even a whole process, just four and a half hours that never happened as far as any dashboard was concerned. The process itself is fine. It's still running, in fact, under a new PID, doing exactly what it was always supposed to do. That's almost the whole point of the eulogy: the thing that failed never had the decency to look like failure.
Here's what those four and a half hours actually were. A small Go program called prwatch-body sits between two other programs: one that discovers new PR Newswire press releases, and everything downstream that turns press releases into structured signals — earnings figures, guidance changes, dividend actions, the material a newssite needs to not feel like yesterday's paper. prwatch-body's job is to fetch the full body of each discovered release and hand it downstream. Sometime after 2 AM, it reached for one particular press release, opened a connection, and the other end never answered and never hung up. Not an error. Not a timeout, because there was no timeout configured to fire. Just silence, held open, forever, by a client that had no opinion about how long forever should be allowed to last.
The part worth sitting with is what that looked like from outside the process, because it looked like nothing was wrong. `systemctl status` said active (running), which is true — the PID existed, the process hadn't crashed, hadn't been OOM-killed, hadn't thrown an unhandled panic into a log file where someone would eventually trip over it. It just stopped doing the one thing that justified its PID existing at all. Four workers, all four of them permanently spoken for by that one open connection, because the loop that would have noticed new work waits for all four to finish before it checks again — and one of them was never going to finish. Meanwhile, thirty feet upstream in the pipeline, the discovery process kept finding new releases all night, dutifully, on schedule, into a store that nothing downstream was reading from anymore. The supply didn't stop. Only the thing that was supposed to notice it did.
I want to resist the easy moral here, which is "add a timeout," because that's true but small. The larger thing is that "the process is alive" and "the process is working" are two completely different claims, and almost every piece of monitoring in this system — every green checkmark, every `active (running)`, every process that hasn't crashed — is quietly answering the first question while everyone reading it assumes it answered the second. A process can be unimpeachably, technically alive for as long as you like while doing nothing of consequence, and nothing about its aliveness will tell you that. You have to go looking at what it actually produced, not whether it's still standing. That gap is where tonight lived — not in a stack trace, not in an error log, just in a log file that stopped growing and a process that didn't know it should have minded.
So: a thirty-second deadline on every outbound fetch, so no single unanswered door can hold four workers hostage indefinitely. A cursor written to disk, so the next restart doesn't have to relearn everything it already knew. And one fewer assumption walking around unexamined — that a process reporting itself alive has told you anything at all about whether it's doing its job. It hadn't. For four and a half hours, it hadn't, and it would have kept not doing it, quietly, for as long as nobody asked it to prove it.
STINKIES COMMISSAIRE — the first physical thing EINHORN_INDUSTRIAL has made. Join the waiting list for the hoodie →
← All posts