Why I built a private AI server instead of renting one
Most people who use an AI assistant today rent it. You open a tab, type into someone else’s model, and pay by the month for the privilege. I did that too, for a while. Then I built one in the house, and I have not looked back. This post is the case for doing it, and an honest account of what it asks of you.
What I mean by “private”
The machine sits at home. The model runs on it. When I ask it something, the question and the answer never leave the building. There is no subscription, no terms-of-service update that changes what it may remember about me, and no product manager deciding that the feature I rely on is being sunset. It is infrastructure, in the plain sense: built once, kept, and mine.
The nearest analogy I have found is the personal computer. In 1995 having one at home was a choice; ten years later it was assumed. I think a private model at home is on the same curve, and I would rather be early and clumsy than late and fluent in someone else’s product.
The problems it was built to solve
I work in IT, which means holding a lot of technical knowledge in my head at once and never quite trusting that I will find the right piece when I need it. Five things kept costing me time, and the build was aimed at each of them.
Information overload. The important note is always somewhere. The problem is that “somewhere” is a folder I have not opened in a year.
Learning bottlenecks. Courses and books are written for everyone, which means they are written for no one in particular. Learning is faster when the explanation starts from what I already know.
Productivity limits. Research, drafting, and documentation eat hours that should go to the work only I can do.
Knowledge fragmentation. Notes, ideas, procedures, and lessons live in a dozen places with nothing intelligent connecting them.
After-hours access. Questions do not keep office hours. A consultant does.
A rented assistant helps with some of these. It cannot help with the ones that depend on knowing my own material, because I am not going to hand my own material to a company that trains on it.
The shape of the thing
Three parts do most of the work. A large language model, the kind with tens of billions of parameters, runs on a GPU in the server. Beside it sits a personal knowledge base, an ordinary folder of Markdown notes that I have kept for years. Between them sits retrieval: when I ask a question, the system first searches my notes, then gives the model what it found along with the question. The answer comes back knowing my context, not just the world’s.
That last part is the whole point. Ask a rented assistant about a project and it knows what the internet knows. Ask this one and it knows what I wrote down last March.
What it actually costs
Not mostly money, though there is hardware to buy. The real price is attention. A private server is a small data centre with a staff of one, and the staff has to learn to run it. Storage fills up. Services need updating. Backups have to be tested, not assumed. The model that was best in spring is second-best by autumn, and someone has to decide whether that matters.
I have come to think of that attention as the point rather than the tax. Every hour spent running the machine is an hour of understanding how these systems work, which is exactly the understanding a rental hides from you. But I will not pretend it is free. If you want an assistant and nothing else, rent one. If you want to own the capability, build one, and budget the evenings.
After giving this some thought, and to speak honestly about what this can really cost: starting smaller than you think is a matter of the philosophy that grounded the idea. I knew every idea that went into the project had to serve the end goal, and it is not easy to get something like this off the ground at the price I started at, which will not make it accessible to many people. There is room to cut hardware. I could have chosen a motherboard with its GPU slots spaced so that I needed only three cards in total, with no wasted slots, and I could have skipped the error-correcting memory. Truth be told, very little about this project would change if I did it again, beyond slimming down some of the hardware and configuration I won’t be using. But the next time I do this I will also be dramatically increasing the capacity of the build, which in my eyes means the next build could very well cost considerably more than the first. Many of the notes I kept through the first phases of the build record its flaws, and how I would approach it differently.
Start smaller than you think
You do not need the server to start. Most of the foundation, the notes, the habits, the first experiments with a small local model, can be built on an ordinary workstation and moved later. I started that way. The server came when the workstation ran out of room, not before.
Why write about it
The blog is the first public thing to come out of the build, ahead of any product or service, because the notes are the part I can share without giving anything away. Everything here will be written from the idea and the experience, not from the configuration. If a post helps you try one thing on your own machine, it has done its job.