From Training to Inference: A Beginner's Map of Modern AI

I spent the last few weeks trying to understand why every AI announcement now sounds like an electrical engineering conference.

NVIDIA is talking about gigawatt AI factories. OpenAI has designed an inference chip with Broadcom. AMD is selling complete racks. Etched is talking about voltage, cooling and cluster memory. Kimi has announced a 2.8-trillion-parameter model with a one-million-token context window.

I understood the software part. Models get better. Products get smarter. What I did not understand was why everyone suddenly needed to own the chip, the rack, the network and, apparently, the power station.

The answer is simpler than the news makes it sound:

AI has become useful enough that the bottleneck is no longer only our ability to design intelligence. It is our ability to deliver that intelligence quickly, cheaply and at enormous scale.

Once I saw the stages separately, the hardware news stopped looking like unrelated noise.

The map

An AI model has a life before it reaches your screen.

Each stage creates a different engineering problem.

Pretraining needs an absurd amount of parallel computation. Post-training needs the model to generate attempts, receive feedback and improve. Inference needs fast responses at a price people can afford. Agents repeat inference many times and add tools, memory, permissions and reliability to the problem.

The industry is not moving from software to hardware. The software has become demanding enough to expose the physical machinery underneath it.

First, four words that make the rest easier

A model is a large mathematical system that has learned patterns from data.

A parameter, or weight, is one of the adjustable numbers inside that system. Imagine an audio mixing desk with billions or trillions of tiny knobs. Training adjusts the knobs. Inference uses their final positions.

A token is a small unit the model reads or generates. It may be a word, part of a word or punctuation. This is why AI companies charge per million input and output tokens.

Context is the information available to the model for the current request: your prompt, conversation history, documents, images or code.

When a company announces a trillion-parameter model or a million-token context window, it is not merely announcing a bigger number. It is creating a storage, memory and movement problem.

Training is where the weights change

Pretraining is the expensive process that teaches a model general patterns.

At a very simplified level, the model receives an incomplete example and tries to predict what comes next. The system measures how wrong the prediction was, works out which weights contributed to the error, adjusts them slightly and repeats this across enormous amounts of data.

The forward calculation is called the forward pass. The process of tracing responsibility for the error is called backpropagation. The number describing how wrong the model was is called the loss.

Training needs so much hardware because it does not only run the model. It must also keep intermediate results for the backward pass, update billions of weights and coordinate thousands of processors doing pieces of the same job.

Those processors have to remain in sync. They need fast memory, fast networking, huge datasets, checkpoints in case something fails, reliable electricity and cooling that prevents the machines from cooking themselves.

A fast chip alone does not solve this. The entire cluster has to behave like one computer.

Post-training turns a predictor into something useful

A pretrained model may know a great deal and still be terrible to use. It may continue your sentence instead of answering the question. It may ignore instructions, choose an awkward format or fail to use a tool correctly.

Post-training shapes the raw model into an assistant.

Humans or automated systems provide examples of good responses. Evaluators compare possible answers. Reinforcement learning rewards attempts that succeed: a coding answer whose tests pass, a maths answer that is correct, or an agent that completes the task.

This creates an interesting change. Post-training is partly a training workload and partly a giant inference workload.

The model may generate hundreds of attempts inside browsers, code sandboxes, games or simulations. GPUs run the model. CPUs run much of the environment around it. This is why the AI machine is becoming a collection of specialised processors rather than one magic chip.

Inference is every time somebody uses the model

Inference means using the trained model to produce an answer.

When you send a message to Kimi, ChatGPT or Claude, you are generally causing inference. The model’s weights are mostly fixed. The system reads your request, calculates a response and returns tokens.

Training and inference are easy to confuse because both run the model. Economically, they are very different.

Training is a large planned project. Inference happens every time a person or agent makes a request.

Training is educating a chef. Inference is every meal the chef prepares afterwards.

A company may train a model periodically. A successful product must serve it continuously. The cost repeats for every prompt, every generated image, every line of code and every agent step.

That is why inference has become the centre of the conversation. Training creates the capability. Inference is where the capability reaches a user and where the bill keeps arriving.

Inference is really two different jobs

The first part is prefill.

The system reads and processes your prompt. A long document, codebase, video or conversation makes this stage expensive. Many input tokens can be handled in parallel, so prefill tends to be compute-heavy.

The second part is decode.

The model produces the answer one token at a time. Each new token depends on what came before it. This makes decode sequential and sensitive to how quickly the machine can fetch the model’s weights and context from memory.

You feel prefill as time to first token: the pause before the answer begins.

You feel decode as tokens per second: how smoothly and quickly the answer streams.

These are not the same problem. A system can begin quickly and generate slowly. Another can take time to digest a giant prompt, then produce the answer very quickly.

This distinction explains one of AMD’s July announcements. AMD described a hybrid system with Cerebras where one type of hardware handles prompt prefill and another handles fast token generation. Instead of pretending one processor is perfect at both jobs, the system separates them. AMD’s Advancing AI 2026 event

The memory wall

I used to assume AI speed was mostly about how many calculations a chip could perform.

It is not.

A processor can only calculate with data it has received. During decode, it repeatedly needs model weights and the stored state of the conversation. If those values cannot reach the compute units quickly enough, the expensive processor sits waiting.

This is the memory wall.

Think of a brilliant chef who can prepare a meal in ten seconds but waits five minutes for every ingredient to arrive from a distant warehouse. Hiring a faster chef does nothing. The supply line is the problem.

Hardware designers use different layers of memory to attack this. SRAM is extremely fast and close to the processor but small and expensive. High-bandwidth memory, or HBM, holds more but is farther away. System memory and storage are larger again, but slower.

Long context makes the problem worse. The system keeps reusable attention information in something called a KV cache. The longer the conversation and the more simultaneous users, the more context memory the provider must manage.

A one-million-token context window is therefore both a model feature and an infrastructure warning.

Then come the network, power and cooling walls

Frontier models rarely fit on one chip. Parts of the model, batches and experts are distributed across many processors.

Those processors constantly exchange information. If one stage is late, the others wait. At this scale, the network is no longer a cable connecting computers. The network is part of the computer.

Then there is electricity.

Switching billions of transistors produces heat. If the heat cannot escape, the chip reduces its speed to protect itself. A benchmark can advertise impressive peak performance while the real machine delivers much less after running continuously.

At data-centre scale, the questions become physical:

• Can the local grid supply enough electricity?
• How much power can enter each rack?
• Can liquid cooling remove the heat?
• How much energy is memory and networking consuming?
• How much useful work does the entire system deliver per watt?

The important unit is moving from peak calculations per second to useful tokens per watt, per dollar and per rack.

This is why the announcements now include cold plates, power delivery, optical links and grid capacity. The companies are not wandering away from AI. They have followed the AI problem down to the wall socket.

Why everyone is building an “AI factory”

NVIDIA’s phrase is deliberate.

A traditional factory converts raw materials and energy into products. An AI factory converts data and electricity into tokens, predictions, generated media and agent actions.

Improving one layer exposes the next bottleneck.

NVIDIA’s Vera Rubin platform is not presented as one GPU. It combines GPUs, CPUs, NVLink switches, network interfaces, storage and data-processing systems in rack-scale designs. NVIDIA’s recent production and infrastructure announcements extend the product boundary again: from chip to rack, from rack to data centre, and from data centre to national power planning. NVIDIA’s Vera Rubin production update, NVIDIA and SK Telecom’s AI infrastructure project

This is also why OpenAI, a company known for models and software, unveiled its own inference processor with Broadcom in June. OpenAI says Jalapeño is designed around its current and future model workloads and will be deployed over multiple generations. If you serve billions of requests, a small improvement to the cost of every request compounds into something enormous. OpenAI’s Jalapeño announcement

The competitive product is becoming the complete production line, not the isolated chip.

What Kimi, AMD and Etched are telling us

Kimi’s K3 announcement describes a 2.8-trillion-parameter mixture-of-experts model. Instead of activating everything for every token, a router selects 16 of 896 experts. This reduces computation per token, but every expert still has to be stored somewhere and reached when selected.

Mixture-of-experts trades one problem for another:

K3 also advertises a one-million-token context window. That increases what the model can hold at once, while increasing pressure on prefill, the KV cache and memory. As of July 25, Kimi says the complete weights are scheduled for July 27, so these are launch claims that still need broad independent deployment and testing. Kimi K3 documentation

AMD’s Helios announcement follows the same system-level direction. The company is combining accelerators, EPYC CPUs, Pensando networking and its ROCm software into a rack-scale platform. The separate prefill and decode plan is especially important because it treats inference as multiple workloads instead of one vague thing called “AI compute.” AMD Advancing AI 2026

Etched is attacking two limits at once. Its Low Voltage Inference approach reduces voltage in mathematical units to control heat and sustain more compute. Its Cluster Scale Memory combines SRAM, HBM and a custom interconnect to reduce the time spent moving data. Etched says it has begun production and is validating rack-scale systems with customers, but its performance figures remain company claims until they are independently reproduced in broad deployment. Etched’s July progress update

Different companies. Different language. Same underlying direction.

The model is colliding with memory, networking, power and latency. The companies are redesigning the whole path from transistor to token.

Agents multiply the problem

A chatbot may receive one prompt and produce one answer.

An agent may interpret a goal, make a plan, search the web, read documents, call an API, execute code, inspect the result, correct itself and ask for approval before continuing.

One request can create dozens or hundreds of inference calls.

That raises demand for cheap reasoning tokens, fast repeated responses, persistent context, CPU environments, identity, permissions, audit logs and security.

It also changes what “performance” means. The fastest model is useless if the agent loops for forty steps, spends ten dollars and still fails the task. The real metric becomes successful work per dollar, with an acceptable wait and a clear record of what happened.

Software did not lose

The hardware boom can make it look as if software has become secondary. That is the wrong reading.

Hardware provides possible performance. Software decides how much of it becomes useful work.

Compilers decide how a model maps onto a chip. Inference runtimes schedule requests. Batching systems keep processors busy. Caches prevent repeated work. Quantisation reduces the memory needed for weights. Agent software manages tools, permissions, failures and state.

A badly scheduled accelerator is an expensive heater.

A poorly designed cache wastes scarce memory. An agent without reliable permissions turns capability into a security incident. A specialised chip without a usable compiler is silicon nobody wants to program.

The new stack is not software versus hardware.

If any one term is weak enough, it limits the whole result.

How I now read an AI announcement

I ask five questions.

Which stage is being improved? Pretraining, post-training, prefill, decode or agent execution?

Which wall is being attacked? Compute, memory, networking, power, cooling, reliability or cost?

Which metric moved? Time to first token, tokens per user, total throughput, cost per token, tokens per watt or task success?

What did the company compare against? “Ten times faster” is beautiful nonsense without the model, precision, batch size, context length and power limit.

Is it actually shipping? A simulation, laboratory chip, customer test and production deployment are four different things.

This does not make every announcement true. It makes the claim legible.

The direction is no longer confusing once you follow the bottleneck.

Training creates capability. Post-training shapes it. Inference delivers it. Agents consume it repeatedly. Each step puts more pressure on memory, networking, power and software. So the product expands from model to runtime, from runtime to chip, from chip to rack, and from rack to data centre.

AI companies are not adding hardware because software stopped mattering.

They are adding hardware because the software finally matters at physical scale.