The night a mount froze everything
The dashboard was still up. That was the part that threw me.
When a system crashes, it at least has the decency to tell you. A stack trace, a restart, a clean gap in the logs with a clear edge. This was different. One night Alix, my self-hosted AI, the one that runs on hardware in my house, went quiet all at once. Not a crash. A freeze.
The health check stopped answering first. (A health check is a tiny endpoint whose only job is to reply “yes, I’m alive” when something pings it. When even that goes silent, you know it’s bad.) The dashboard I keep open loaded its shell and then hung, spinner turning, waiting for numbers that never arrived. A reply she’d been streaming to me stopped mid-sentence and stayed there, half a thought, cursor blinking, going nowhere.
Everything stopped at the same instant. And the process was fine. Up, running, using almost no CPU, doing precisely nothing. From the outside it looked dead. From the inside it was alive and completely stuck.
The cause was almost insulting
Here is the whole villain: a network file share had stalled.
A share is just a folder that lives on another machine and gets “mounted” so it looks like a local directory, you read and write it as if it were sitting on your own disk, but every access is really a quiet trip across the network. Usually that trip is fast enough that you forget it’s happening. That’s the trap. When the far end goes unresponsive, the folder doesn’t error out. It just… waits. A read that normally takes a millisecond now takes forever, and forever is a long time to hold a lock.
That, by itself, should have been survivable. A slow folder is annoying, not fatal. What made it fatal was where the reading happened.
Some of Alix’s file-browsing tools, the ones that list a directory or open a file so she can look at what’s there, were doing their disk reads directly on the event loop.
One worker, one rule
If you’ve never had to care about an event loop, here’s the whole idea in one image.
An async system is one very fast worker juggling thousands of tasks. Not thousands of workers, one. It handles a request until that request has to wait on something slow (a database, the network, a disk), and in that waiting moment it sets the task down and picks up another. It’s constantly switching: start this, park it, advance that, park it, come back. Done well, a single thread can keep thousands of conversations moving at once and none of them can tell they’re sharing.
The whole trick depends on one rule: no single task is ever allowed to actually block. Every wait has to be the polite kind, “wake me when the data’s ready”, that hands the worker back so it can go do something else in the meantime.
A blocking call is the rude kind. It doesn’t hand the worker back. It seizes the one thread and holds it until it’s good and finished.
So picture that worker mid-juggle, and one of the things it picks up is a synchronous read against a mounted share that has silently gone dead. The read doesn’t return. The worker is now standing there holding a task that will never complete, and it cannot set it down to go tend the others. The health check that just needed a half-millisecond to say “I’m alive”? Behind it in line. Your streaming reply? Behind it. Every other task in the entire system, all thousands of them, waiting on one worker that is waiting on one folder.
One blocking call is a single point of failure for the whole process. Not for one feature: the whole thing.
The fix is boring: and boring is usually a good sign
You do not make the disk faster or the mount more reliable. You cannot promise either. You make it so a slow disk can only ever hurt itself.
The fix was to move all filesystem work off the event loop and onto a thread pool, a small crew of separate worker threads whose entire job is to go do the blocking, patient, dangerous things so the main worker never has to. Now when a file-browsing tool reads a directory, that read happens on one of the crew. If the folder is hung, that crew member blocks indefinitely, and only that thread does. Meanwhile the event loop never touches the disk, never blocks, and keeps switching between everything else exactly as before.
A stalled share can now stall the thread doing that read, at worst a handful of pool threads, if several reads hit the dead mount at once, instead of the whole process. The pool holds a fixed number of threads, so even that worst case is bounded. The health check answers. The dashboard fills in. Streaming replies keep coming. The blast radius went from “the entire system” to “the reads unlucky enough to touch the dead mount.”
The lock
Fixing a bug is only half the job. The other half is making sure the fix can’t quietly walk back in six months from now when someone (possibly me) writes a new file tool in a hurry and forgets the rule.
So there’s a regression test, and it’s blunt on purpose. It reaches in and patches directory listing so that it sleeps, deliberately turns a folder read into a slow, blocking one, simulating exactly the hung mount that started all this. Then, at the same time, it kicks off a completely separate task and asserts that the separate task still finishes on time.
If filesystem work is properly off the loop, the sleeping read stalls its own thread and the other task sails through. The test passes. If anyone ever puts blocking I/O back on the event loop, the fake stall grabs the shared worker, the innocent task misses its deadline, and the test fails, before the change ever ships. The rule is no longer something I have to remember, the test enforces it now.
Check the plumbing before you blame the mind
Here’s the part I keep relearning, and the reason I’m writing this down.
When Alix “breaks,” the instinct is to suspect the interesting thing: the model, the reasoning, the personality. It almost never is. This freeze wasn’t the model. Neither was a bug where a stream with no timeout could hang open. Neither was a stale cached page served up as if it were live. Neither was a batch of old search results getting re-served as fresh. Every one of those got reported to me, in the moment, as some version of “she’s acting broken.” Every one of them was plumbing.
An honest system with a frozen event loop is indistinguishable, from the outside, from a dead one. It isn’t lying to you and it isn’t confused, it’s stuck behind one blocking call, and it has no way to tell you so, because the part that would tell you is stuck in the same line. The failure hides in the least interesting layer precisely because that’s the layer nobody’s watching.
So the discipline is boring, and that’s the point: when she goes quiet, check the plumbing first. Every failure I’ve actually traced has been plumbing, not the mind. That night it was a folder that stopped answering: and she’d been waiting behind it the entire time.