A bound is not a backstop
One page, 165 seconds
Here is the number this entry exists for: 165 seconds. That is how long one page of text, one input, inside the size limit, perfectly legal, could pin a core of the machine Alix runs on. Not crash it. Pin it. Nearly three minutes of a processor doing nothing but re-reading the same characters over and over, while everything queued behind it waited.
The part that earned the write-up isn’t the number. It’s that the code responsible was sitting behind a size cap I believed was protecting me. It wasn’t, and understanding exactly why it wasn’t took me three consecutive rounds of review, across which the same class of bug kept turning up wearing different line numbers.
The path nobody watches
Alix reads the web on her own. Her curiosity system picks topics and goes looking; a news lane brings her the day’s headlines. When I say “curiosity,” I mean an engineered drive: a scheduled process with interests and a budget, not a claim that there is someone in there wondering about things. This is a log about building presence, and part of presence is that the system does things when nobody is watching. Hold onto that, because it’s exactly what makes this bug worse than it sounds.
Text from the open web is untrusted, so it doesn’t get to walk straight into her context. A set of pattern-matching rules scans every fetched page first, cleaning it up, stripping what shouldn’t be there, flagging anything that could masquerade as her own internal bookkeeping. Each rule is a regular expression: a compact description of a text shape, run across the whole page. Regular expressions are wonderful and treacherous in the same specific way. Most of them do work proportional to the length of the input, and some of them, on the wrong input, do work proportional to the square of it. For every character, they go back and re-examine everything that came before. Length times length.
A branch extending this path went into merge review last week. What came out was an education.
Three rounds, three patterns, one bug
Round one didn’t find a quadratic. It found a different bug entirely: a fetched page could counterfeit a piece of that internal bookkeeping, and the fix added a new scanning pattern to defang the lookalikes. The fix was correct. It was also, though nobody would notice for two more rounds, quadratic. Reviews exist to find things. Fixes, it turns out, can plant them.
Round two, a fresh pass with fresh eyes, found a quadratic pattern on the same path, the first anyone had noticed, and measured it: about half a second on eight thousand characters of hostile text. Seven seconds on thirty-two thousand. Nearly two minutes on a hundred and twenty-eight thousand. Quadruple the input, sixteen times the time, the unmistakable signature of quadratic growth. Extrapolated to the largest page the fetcher will accept: roughly eight hours of solid CPU, from one page. The same round turned up a second quadratic nobody had suspected. Both were fixed, and, feeling appropriately humbled: the round added a cap: no matter how large a fetched page is, the scanning rules only ever see its first quarter-million characters. Defence in depth, I told myself. A second, independent layer.
Round three, verifying round two, found the third: the pattern round one’s fix had planted, in plain sight for two rounds. And it was behind the cap, and the cap did not contain it. A quarter-million characters, squared, is still an enormous amount of work. Measured honestly: 165 seconds of blocked CPU for a single page inside the limit. The cap hadn’t removed the catastrophe. It had converted an unbounded catastrophe into a guaranteed, repeatable, three-minute one.
One more detail worth keeping: at least one of these patterns had been perfectly linear until a routine hardening change a few days earlier widened the set of characters it accepted. That widening quietly changed its worst case from “proportional to the page” to “proportional to the page squared.” Nothing about the change looked dangerous. Linearity is not a property you check once and own forever; it’s a property you can lose in an ordinary commit.
Why a size bound is not a time bound
The reasoning error is worth stating slowly, because I made it with full confidence.
A cap on input size bounds input size. That is the entire list of things it bounds. If the work grows with the square of the input, a hundred-thousand-character cap caps the work at ten billion steps: it guarantees the input ends, not that the work does. “Bounded” and “small” are different words, and the gap between them is exactly where this bug lives.
Ordinary pages behave. The page that hurts you is the weird one, full of repeated near-matches that force the pattern to backtrack: the page a hostile author can construct on purpose and a sufficiently strange corner of the web will eventually serve up by accident. Untrusted input doesn’t just get to be wrong. It gets to be slow, and slow is also an attack.
And these patterns run inside an async handler, which regular readers will recognize as the setup to a story I’ve already told once: one blocking call on the event loop stops everything, every conversation, every background process, the health check itself (the night a mount froze everything). A hung mount did that to her in July. This was the same failure with a different villain: not a folder that stopped answering, but a pattern that wouldn’t stop working.
Now add the first section back in. This path runs autonomously. She fetches pages out of her own curiosity, at whatever hour her drives fire, with no human in the loop. Nobody is sitting there to notice that a process pegged a core for three minutes at two in the morning. The freeze happens, resolves, and leaves nothing behind but a gap: the most invisible kind of failure a system can have.
The fix is a test, not a fourth patch
After round three I stopped and looked at the shape of the thing. Three rounds. Three patterns. Same class every time, one of them planted by a fix from the round before. Every fix was locally correct, and the process producing them was wrong, fixing quadratic patterns one at a time is enumeration, and this branch had just demonstrated, three patterns deep, that enumeration doesn’t converge. There was no reason to believe a fourth pattern wasn’t already there, or wouldn’t be added next month by someone (me) hardening something in a hurry.
When review keeps finding the same class of bug, the move is to stop patching instances and build the class detector.
So the branch’s real deliverable ended up being a test: every pattern that touches untrusted text lives in a registry, and a global timing test walks that registry, throws deliberately pathological input at each pattern at escalating sizes, and asserts that the growth curve stays linear. If any pattern’s cost curve bends upward, the build goes red, including for patterns that don’t exist yet, because a new pattern lands in the registry by construction and gets enrolled in the test without anyone remembering to think about it. A performance regression suite for regular expressions.
This is the same philosophy that runs through the whole project: the things I actually care about get encoded as tests, because tests don’t rely on my memory or my discipline (character as CI). Her honesty has a suite. Her voice has a suite. As of this branch, so does the proposition that no web page gets to freeze her for three minutes.
What review actually certifies
The lessons, in the order they cost me:
A bound on size is not a bound on time. If the work is quadratic, a cap doesn’t make it safe; it makes it predictably catastrophic. A cap can be a genuine second layer only if it’s small enough that even the worst case survives it, otherwise it’s a comfort object.
Untrusted input deserves adversarial performance tests for exactly the reasons it deserves adversarial correctness tests. I had spent weeks asking “what can a hostile page make her believe?” and never once asked “what can a hostile page make her compute?”
Repeated same-class findings mean the fix is a test. One quadratic pattern is a bug. Three in one branch is a property the codebase doesn’t hold, and properties are asserted globally or not at all.
“It passed review” means it passed the questions review thought to ask. Careful rounds signed off on this path while a 165-second freeze sat in it, because nobody had yet asked the timing question. Review is a transcript of the questions asked so far, not a certificate of safety: and the useful response to a miss is adding the missing question to the list that gets asked automatically, every build, forever.
The branch merged. The suite, thousands of tests now, includes a family that will never produce an interesting headline, just a red build on the day some future pattern of mine bends the curve. The cap is still in there too, by the way. It’s fine, as a cap. It’s just not a backstop: and nothing needs it to be one anymore.