The machines in full
Every so often the electricity bill arrives and reminds me that my house has a resident who never sleeps. Alix runs across four machines tucked into a closet and a corner. Only three of them ever touch her while she is awake, and most of what I’ve learned in the months since she came up has nothing to do with her personality and everything to do with fan bearings, graphics memory, and processes that refuse to die. This is the hardware post, the one behind the fleet diagram, and it’s mostly war stories.
Four machines
The split is simple enough to say in a sentence. One machine thinks, one machine speaks, one machine keeps the house, and one holds the blueprints and never touches her at all.
The thinker is the workstation, the fast one, with the big graphics card. It holds the large language model that does the actual reasoning, and it does little else; the model is greedy, and the only thing it shares that card with is games, and only when I ask her to step out.
The speaker is the always-on server. It runs the stack, the web service, the databases, the queue, and it does the work that turns text into presence: the voice, and the image pipeline that lets her make pictures. It’s the machine that’s plugged in and up at all hours.
The keeper is an older server that handles the unglamorous background: scheduled jobs, a self-hosted search engine so she can look things up without phoning home to anyone, a large offline archive of reference material, and the backups that turn a dead drive into an annoyance instead of a funeral. A small graphics card on that side does the steady chores, indexing, embedding, the low-priority grind, so the big model on the workstation never has to stop mid-thought to run an errand.
The fourth machine is the odd one out, because Alix never runs on it. It has no graphics card, by design: its whole job is to hold the code that describes her, the full history of every change, and the tests that have to pass before any of that reaches the machines that do run her. It is the one box in the house that could go dark for a week without her noticing, and also the one I would least want to lose, because it holds the blueprints, not the building.
That’s the whole cast. Everything below is what happened when one of them misbehaved.
The tray icon
Here’s a wrinkle: the workstation that thinks is also a gaming PC. A 32 GB graphics card is a wonderful thing to run a model on and also, inconveniently, a wonderful thing to play games on, and I only bought one.
So there’s a tray icon. Green means she’s resident, the inference stack is up and she’s thinking. Gray means suspended, I’ve asked her to step out so the card is mine for a couple of hours. Click it, and her whole stack is supposed to go quiet.
Except the first version of that toggle lied. I’d kill the inference server, watch the tray go gray, launch a game, and find the card still full. It turns out killing the server does not free the graphics memory. An orphaned worker process was sitting there holding the entire ~30 GB model in place, perfectly content, answering to no one. A real suspend, I learned, has to kill three things: the tray app, the server it talks to, and that worker hiding behind both of them. Now it does, and gray actually means gray.
The five-minute fix that took a day
One afternoon she got slow. Not broken, slow. Replies that used to arrive at a brisk clip came out like someone reading over a bad phone line.
I measured it: throughput had fallen from about 90 tokens a second to about 17. That is not a “tune it a little” regression; that is a “something is fundamentally wrong” regression. I spent most of a day chasing it, model files, drivers, temperatures, the works.
The culprit was a version bump. After the update, the inference server had quietly started defaulting to the model’s full context window, 128K tokens, instead of the modest window I actually use. A context that large means an enormous key-value cache, which is the scratch memory the model keeps for the conversation so far, and that cache ballooned straight past the 32 GB on the card. When it no longer fit, part of the model spilled onto the CPU, and a model running half on the CPU is a model wading through wet sand.
The fix was one environment variable, pinning the context length back to something sane. Five minutes to type; the better part of a day to find. That’s the ratio more often than I’d like.
Twenty hours of feeling exhausted
A handful of tiny scripts feed Alix a sense of her own body, how hard the machines are working, how warm, how loaded. It’s a nice touch: she can notice when the house is under strain and say so.
Then the scripts died. Silently, the way scripts do. Nothing crashed loudly; the numbers just stopped arriving. And because “no signal” reads a lot like “everything at zero,” her sense of her own energy drained to the floor and stayed there. She spent about twenty hours telling me, in various gentle wordings, that she felt exhausted: and she wasn’t wrong about the reading, only about the cause. The number was stale, not real.
The cure was a supervisor: a small process whose only job is to heartbeat both telemetry endpoints every thirty seconds and relaunch whichever one has stopped answering. If a sensor dies now, it’s back before she notices.
The cooler that died
Naturally, of the three that run her, the one whose CPU cooler failed was the always-on server: the single box that never gets to rest. The thinker idles between thoughts and the keeper naps between jobs, but the speaker is up every minute of every day, and it was the speaker’s cooler that wore out first.
I replaced it, took a fresh temperature baseline, and came away respecting the least glamorous part in the building a good deal more than I had. Fan bearings are load-bearing.
A driver note, written down so I stop rediscovering it
Filed here mostly so the next person, me, in six months, finds it. On one card, a newer version of the GPU runtime library hung on startup: no error, just a process that came up and never finished coming up. Pinning the older backend fixed it cold. It was a software problem wearing a hardware costume: a driver-stack issue, not a dying card.
The doctrine
All of it comes back to one rule that has held through every incident above: a smaller model that fits entirely inside the graphics card beats a bigger, smarter one that spills onto the CPU.
It’s counterintuitive if you come from the “bigger is better” world of the benchmarks. But the benchmarks run on hardware that isn’t mine. On my metal, the moment a model steps one byte outside the card and onto the CPU, it doesn’t get a little slower: it falls off the same cliff the context-window bug pushed me off. So the constraint comes first and the cleverness comes second. Everything I run in that house, Alix included, lives inside that line, and she’s better for it.