Why everyone is suddenly talking about inference chips

For a few years, the AI hardware story was one sentence long: buy Nvidia GPUs, as many as you can, and apologise to your finance director later. That story hasn't ended — but as of early 2026 it has forked, and the fork is the most important thing happening in computing hardware. The industry has realised that the chips best suited to building AI are not necessarily the chips best suited to running it.

Training versus inference, in one minute

Training is the once-per-model marathon: months of number-crunching across thousands of chips to set a model's billions of internal weights. Inference is everything after — each question answered, each email drafted, each image described is one inference pass. Training happens once; inference happens every time anyone uses the thing, forever.

Early in the AI cycle, training dominated spending because everyone was building models and hardly anyone had users. That has inverted. Mature AI products spend the overwhelming majority of their compute budget on inference, and analysts' estimates as of early 2026 generally put inference at well over half of all AI compute demand, still climbing. Training got the headlines; inference gets the invoice.

Why the same chip isn't ideal for both

Training rewards raw parallel throughput: you can batch enormous amounts of work and nobody minds if any single calculation takes a while. Inference serves impatient humans. It cares about latency — the gap before the first word appears — and about cost per token at scale. And its defining bottleneck is usually not arithmetic at all but memory bandwidth: generating each token means hauling the model's billions of weights past the processor, again and again. The compute units of a big GPU often sit partially idle during inference, waiting for memory. You are, in effect, paying for a Formula 1 engine to do the school run.

Training happens once. Inference happens every time anyone uses the thing, forever.

The runners and riders

Nvidia, defending. Nvidia saw this coming and has tuned recent architectures heavily for inference — lower-precision number formats, disaggregated serving, and software that squeezes more tokens per watt. Its real moat remains CUDA, the software ecosystem a decade of AI engineers grew up on. Nobody gets fired for buying Nvidia; plenty of CFOs now ask if you must.

The hyperscalers, self-catering. Google's TPUs are the veteran example — designed in-house precisely because Google could see its future inference bill. Amazon runs Inferentia and Trainium; Microsoft has Maia; Meta builds MTIA for its recommendation and ads inference. The logic is brutal arithmetic: at hyperscale, shaving cents per million tokens justifies a chip programme. Every token they serve on their own silicon is margin reclaimed from their most important supplier.

The specialists, sprinting. Startups such as Groq (with its deterministic, SRAM-heavy LPU) and Cerebras (with wafer-scale chips) attack the memory bottleneck head-on and demonstrate token speeds that make GPUs look sleepy. Their challenge isn't physics; it's economics and ecosystem — persuading developers to leave the CUDA comfort blanket, and manufacturing at a scale that matters.

And in your pocket. The same split is happening in miniature on consumer devices, where NPUs in phones and laptops now run capable small models locally — the shift we unpacked in Small models, big shift. A meaningful slice of tomorrow's inference won't touch a data centre at all. Some of that consumer silicon, incidentally, is quietly built on an open instruction set — see RISC-V for normal people.

Why you should care even if you never buy a chip

Three reasons. First, price: inference cost per token has been falling steadily, and specialised silicon is a big part of why; that decline flows straight into what AI software costs you, a dynamic we traced in our AI pricing analysis. Second, speed: cheap, fast inference is what makes agents, real-time translation and voice interfaces feel usable rather than laggy. Third, power: inference at scale is now a grid-planning problem, and in the UK it collides with a connections queue we examined in Data centres and the grid.

The likeliest outcome, as of early 2026, is not a dethroning but a diversification: Nvidia keeps the frontier and the flexibility, hyperscalers serve their own bulk traffic on their own silicon, specialists carve out the speed-obsessed niches, and a growing share of everyday inference migrates to the device in your hand. The monoculture is over. For everyone who pays for AI by the token, that's excellent news.