- It's all about AI, these days...
Five years ago, the server closet was a punchline. Companies were ripping racks out of back offices, canceling colo contracts, and shipping everything to a hyperscaler. The pitch was simple: stop babysitting hardware, pay for what you use, scale on demand.
Now the same companies are hauling machines back in. Not all of them, and not all workloads, but enough that it has a name — repatriation — and enough that CFOs are asking engineering why the invoice from a cloud provider looks like a second payroll.
One workload is worth watching: AI inference. That’s the thread this piece follows, because it’s the clearest case of a company moving a job out of the cloud and finding that the closet it walked away from is suddenly the best place to run it.
The Invoice That Started the Conversation
The repatriation conversation almost always starts with a bill. A team ships an AI feature, usage climbs, and the monthly line for GPU inference stops looking like infrastructure and starts looking like cost of goods sold. The numbers aren’t small. They aren’t one-off, either.
Vendors building hardware for on-prem LLMs have built their pitch around exactly this moment: the one where the invoice forces the question. Andreessen Horowitz laid out the broader economic case years ago, arguing that scaled software companies carry a meaningful public-cloud tax on their market cap and that bringing infrastructure in-house can roughly halve the bill. That argument used to apply mostly to databases and storage. Now it applies to the GPUs doing inference every time a user clicks a button.
Finance notices this before engineering does. The invoice is itemized. The usage graph only goes up. And the question that follows is the one every repatriation story turns on: does this workload need to live where it lives?
Why the Cloud Stopped Being the Obvious Answer for Inference
Three things changed under the hood while nobody was looking.
- Utilization is terrible. Rented GPUs sit idle most of the time, and you pay for the reservation anyway. When a workload is steady and predictable, paying hourly for a resource you use continuously is the worst possible pricing model.
- Egress is a tax on leaving. Pulling data out of a hyperscaler to process it somewhere else costs real money per gigabyte, and inference pipelines move a lot of data. The pricing is designed so the cheapest place to compute is wherever the data already lives.
- Data residency got teeth. European regulators have grown pointed about where model inputs and outputs physically sit, and “it’s in our account on a US provider” is no longer a comfortable answer for a lot of legal teams.
None of this makes the cloud wrong. It makes a specific job — running a model you own, against data you own, for users whose queries you can forecast — a bad fit for rental pricing. That’s the job coming home.
What Coming Home Actually Looks Like
The server closet of 2026 is not the server closet of 2012. Companies aren’t rebuilding general-purpose data centers. They’re buying small, dense GPU systems sized for one workload and installing them where the data and the users already are — a rack in an office, a few nodes in a colo cage, a ruggedized box at a factory.
The operating model changes with it. You’re not replacing a cloud team with a hardware team. You’re adding one physical system — or a small handful — to an otherwise cloud-native stack. The database still lives wherever it lives, the web tier still autoscales, and the GPUs that were burning cash in a rented region now sit in a box you own, running at the utilization you need.
The Closet Isn’t a Nostalgia Trip
The companies pulling hardware back in-house aren’t doing it because they miss the smell of a server room. They’re doing it because a specific workload — an AI model serving predictable traffic against sensitive data — has pricing physics that reward ownership. The BizTech repatriation explainer frames this well: not an ideological swing back to the data center, but a sorting exercise, workload by workload, with regulated industries and edge sites leading the way.
The server closet came back because one job finally justified it. Expect more jobs to follow — but only the ones where the math and the data agree.
ItIt