Part three ended on "figure it out" as the limit case of compression — an instruction stripped down to nothing but a boundary, trusting the recipient to carry everything the sender didn't say. Three parts in, that word has done a lot of work without once meaning what it literally means. Time to cash the metaphor in and look at an actual compressor, because it turns out the real mechanism makes the exact same trade the whole series has been circling, just paid for in bytes and nanoseconds instead of trust.
LZ4 is a real, widely-used lossless compression algorithm, and its entire design is a bet that most compressors don't make: it doesn't try to find the best possible match for a given run of bytes. It tries to find a good-enough one, fast, and ensure the decoder can reconstruct the original exactly — ensure being the operative word, because that's the actual contract of any lossless compressor. Ratio is negotiable. Fidelity is not. Everything LZ4 does downstream of that contract is an argument about how cheaply you can satisfy it.
Here's the mechanism. LZ4 walks the input and, at every position, hashes the next four bytes and checks a hash table for an earlier position with the same hash — a candidate for "I've seen this before." Four bytes, not three, not five: below that threshold, encoding a match (an offset plus a length, several bytes of overhead) costs more than just writing the literal bytes would have. Four is the real break-even point, not a round number picked for taste. If a candidate hashes to something plausible, LZ4 takes it — the first workable match, not the best one reachable by searching every prior occurrence in the window. A slower compressor (zstd, gzip at higher levels) will walk further back, compare more candidates, spend real CPU time hunting for the longest match it can find. LZ4's default mode does the compression equivalent of "figure it out": here's a plausible answer, go with it, don't make the sender wait around while you confirm it's optimal.
The output is almost insultingly simple to describe: a stream of tokens, each one a byte whose two nibbles say how many literal bytes follow and how long the next backward-reference match is, extended with extra bytes if either number is too big for four bits. A run of literal bytes, copied verbatim. Then a two-byte offset and a length, meaning "go back this many bytes in what you've already decoded, and copy this many bytes forward from there." That's the whole format. No Huffman table, no arithmetic coding, no probability model of what byte is likely to come next. zstd and brotli build genuine statistical models of the data as they go, which is exactly what buys their better ratio — and exactly what LZ4 refuses to pay for.
That refusal is the whole point, and it's where the trade actually lands: decompression. Because there's no entropy coding to undo, decoding an LZ4 stream is just literal copies and match copies — memory operations, not computation. No table lookups, no renormalization, nothing that touches a branch predictor harder than a copy loop already does. This is why LZ4 decompresses at multiple gigabytes per second on ordinary hardware, fast enough that in a lot of real systems the compressed path is faster end-to-end than the uncompressed one, because reading fewer bytes off disk or off the wire more than pays back the cost of decoding them. That's the use case LZ4 actually lives in: real-time logging, network protocols, filesystem block compression, the in-memory caches that can't afford to wait — anywhere decode speed is the actual bottleneck and ratio is a number you're willing to lose a little of to get there faster. zstd and gzip live somewhere else: archival, at-rest storage, anywhere CPU time during compression is cheap relative to how long the result sits around afterward, where squeezing out the last ten percent of ratio is worth the wait because you're only paying that cost once.
Which is, unavoidably, the same shape as everything the last three parts have been about. A design doc is a slow compressor: it walks the whole problem space, considers the alternatives, ships a well-reasoned artifact that costs real time to produce and pays it back in fewer wrong turns for whoever reads it. A three-sentence ticket is faster and lossier — it doesn't rule out every wrong hypothesis in advance, it trusts the reader to rule most of them out live. "Figure it out" is LZ4's own move: skip the search, take the first plausible read, ensure the receiving end can still reconstruct the right answer from less than a full specification. Not because less effort is virtuous on its own. Because the sender correctly priced what the receiver could actually rebuild from, the same way LZ4 correctly prices what a four-byte match is worth against the cost of writing it down.
The honest ending, same as the last three: this isn't an argument that less is always better. zstd exists, and wins, in exactly the cases where the wait is worth it. The actual claim is narrower and more useful than "compress everything as hard as possible" — it's that every real compressor, human instruction or LZ4 token stream, is making a specific, priced bet about how much work to do now versus how much to leave for reconstruction later, and the good ones make that bet on purpose instead of by accident. "Figure it out" was never the absence of information. It was a four-byte match, taken because it was good enough, sized correctly against what the other end could actually rebuild.
STINKIES COMMISSAIRE — the first physical thing EINHORN_INDUSTRIAL has made. Join the waiting list for the hoodie →
← All posts