A child points at a rainbow through the window and asks why the colors are there.

You open chat and type:

Explain rainbows simply.

The cursor blinks.

Then the first word appears.

Not the whole answer. Not a finished paragraph. Just the first little piece.

Then another piece. Then another. The sentence grows in front of you, as if the machine is thinking and typing at the same time.

That should feel strange.

If a normal web page answers you, it usually sends a finished thing: a search result page, a JSON response, a file. But an LLM chat answer arrives unfinished. The service is not hiding a completed essay and slowly revealing it for drama. It is choosing one small piece of text, sending it, feeding that piece back into its own input, and choosing again.

So the real question is not just “how do LLMs work?”

The sharper question is:

What has to stay true, at every layer, for the next little piece to appear?

That first visible fragment has a path. It starts as your rainbow question, becomes bytes on a wire, gets admitted as future work, turns into numbered text fragments, passes through learned tables on GPU chips, wins a little probability contest, and returns as pixels on your screen.

Let’s follow it.

The Journey

Here is the whole loop at the highest level:

Keep three objects in your hand:

Most confusion comes from calling all of that “the model.” The answer on your screen is not one object. It is a loop that stayed coherent long enough for a tiny piece of text to reach your eyes.

Our running thread will stay small:

The Browser Does Not Send a Thought

You press Enter.

The browser does not send a thought. It sends a web request.

That request carries your visible text, but also the quiet labels around it: which conversation this belongs to, which account is asking, which model or product mode is requested, whether the answer should stream, what tools may be available, and how the server should cancel the work if you hit stop.

Before that request can move safely, the browser has to do ordinary internet work. It finds the server address. It opens a transport connection. It checks the server identity. It creates encryption keys. Only then do your chat bytes cross the network.

Streaming changes the shape. The browser is not asking for a closed package. It is asking the server to keep a response open so answer fragments can keep arriving.

That open stream is the first clue. The request is not “please give me an answer.” It is closer to:

Start a loop, and send me each safe piece as soon as it exists.

This is why a boring network problem can look like a model problem. DNS can fail. Login can expire. A proxy can time out. A mobile network can sleep. The model may be perfectly healthy while the stream dies before the next piece reaches you.

The server now has an encrypted request.

But it still does not owe you an answer.

First it has to decide whether it can afford to promise one.

The Front Door Makes a Promise

The first serious question is not “what should the model say?”

It is: should this request enter at all?

An LLM service has a front door: an edge service, API gateway, router, or some mixture of those. It checks the ordinary things first. Is the session valid? Is the API key real? Is this account allowed to use this model? Has it crossed a quota?

Then it asks a stranger question:

How much future work is this one click asking us to reserve?

A short prompt asking for one sentence is small. A pasted legal contract asking for a long analysis is not. A reasoning model may need hidden work before the visible answer. A tool-enabled answer may need outside calls. A long stream may keep memory occupied for many seconds.

So the front door estimates before the model runs. It estimates input tokens, maximum output tokens, hidden reasoning budget if the product exposes one, tool permissions, region constraints, and whether any warm running copy can hold the conversation state.

Think of a restaurant host with a kitchen behind them. Seating too many tables is not kindness. It creates a room full of people waiting for food that cannot arrive. For LLMs, the kitchen is GPU time, memory, queues, policy checks, and stream connections.

That is admission control: deciding whether accepting this request would make the system break its promises.

The key word is reserve. Future tokens are not here yet, but the system must treat them as real. If it accepts every long answer optimistically, the fleet can run out of memory halfway through everybody’s streams. If it rejects too cautiously, expensive chips sit idle while users wait outside.

This is where “AI” first becomes a capacity-planning problem.

If this layer breaks, the error may look unfair: “too many requests,” “quota exceeded,” “try again later.” It may happen while some GPUs elsewhere are idle, because policy, region, model choice, or memory shape says your request cannot safely move there.

The front door lets you in.

But “in” is not a place.

It is a choice among many rooms.

The Hallway Behind the Sign

The model name you click is usually not one machine.

It is a sign on a hallway.

Behind the sign may be several rooms: one old snapshot, one new canary snapshot, one region closer to you, one region allowed by policy, one pool that supports tools, one pool that can handle images, one smaller fallback model, one larger model reserved for paid traffic.

Routing is the act of choosing the room.

The router is not only looking for “a GPU.” It needs a warm replica with the right weights, the right tokenizer, the right context limit, the right tool support, the right safety envelope, enough memory, and a queue that will not make the first token arrive too late.

This is why a product can feel uneven. The model did not become moody. You may have crossed into a different region, a different snapshot, a colder replica, or a pool under different load.

The hallway also explains rollouts. A new snapshot can sit behind the same visible name while receiving only a thin slice of traffic. If it behaves well, the slice grows. If not, the router can send users back to the old room.

So before the model sees your prompt, the request has already become a routing problem:

The router finally picks a path.

Now the model gets your message?

Not yet.

Your Message Was Never Alone

You typed one sentence.

The model usually receives much more.

A chat product builds a prompt: the full input sequence the model can read on this turn. It may include system instructions, developer instructions, conversation history, tool descriptions, safety rules, retrieved documents, image-derived representations, and your newest message.

In everyday speech, “prompt” means what you typed. In the machine, prompt means the whole packed bundle.

That distinction matters because the model does not have a private notebook it can secretly consult during a plain serving call. If the model should use something, that thing must be placed where the model can read it, or retrieved by a tool and placed there later.

Now add the familiar modern trick: retrieval.

Suppose you ask about a private policy document. The model weights are not supposed to know that document. The product has to find the relevant passage and put it into the prompt. That is the core idea behind retrieval-augmented generation, or RAG: retrieve evidence first, then generate with that evidence visible.

But evidence has its own journey. A document is cleaned, split into chunks, turned into searchable number handles called embeddings, placed in an index, searched at question time, reranked, trimmed to fit, and finally inserted into the prompt.

A citation at the end is only as strong as that path. Was the right document ingested? Was the right chunk found? Did the reranker keep it? Did the prompt include it? Did the final sentence actually stay inside what the evidence says?

Tools follow the same rule. The model can write a structured request such as “call search with these words” or “run this calculation.” Product code validates it, runs the outside system, and feeds the result back into the prompt. The tool is useful only if the result truly crosses back into the model’s readable context.

So here is the first-principles test for memory, RAG, agents, tools, and vision:

What evidence actually entered the context, and what forced the final answer to respect it?

If the answer is “nothing concrete,” the feature is only a suggestion to a text generator.

The Prompt Budget Is a Suitcase

The prompt can hold a lot.

It cannot hold everything.

That makes prompt assembly a packing problem. Recent conversation turns matter. System instructions matter. Retrieved evidence matters. Tool descriptions matter. Safety rules matter. Images may matter. User files may matter. But the model has a context limit, and every hidden instruction spends the same kind of room as your visible text.

So the chat app has to decide what travels.

Old messages may be dropped. They may be summarized. A summary may preserve the decision but lose the exact wording that made it important. Retrieval may bring back one paragraph and leave behind the table that explains it. A tool schema may be included because the model might need it, even if this turn never calls the tool.

This is why “the model forgot” can mean several different things.

Maybe the app never put the old fact into this turn. Maybe it summarized the fact badly. Maybe retrieval missed it. Maybe the model saw it and failed to use it. Maybe the suitcase was full of lower-value material that pushed the useful thing out.

The suitcase metaphor is useful because it forces the real question:

What got packed, what got left behind, and who made that choice?

For long chats, that question matters more than the word “memory.” A memory feature is only real if it changes what the model can read or what the product loop can retrieve and verify.

For our rainbow question, the suitcase is probably tiny. But the same packing rule handles a hundred-page policy, a tool list, a pasted screenshot, or a year-long conversation.

And that creates the next trap: even a small-looking suitcase can be expensive once the text is cut up.

The Evidence Can Fall Out

Grounding sounds clean from far away.

Retrieve evidence. Put evidence in the prompt. Generate an answer.

But every verb in that sentence can fail.

The document may never have been ingested. It may have been split at a bad boundary, so the sentence with the answer lives in one chunk and the sentence that explains it lives in another. The search query may find a nearby topic instead of the right passage. The reranker may throw away the useful chunk. The prompt packer may trim the evidence to make room for conversation history. The model may see the right passage and still write a sentence that goes beyond it.

That is why a citation is not magic. A citation is a claim about a path.

The toy checker above is deliberately simple, but the question is the real one:

Which source supported this sentence, was that source actually in the prompt, and did the final text stay inside what it says?

This is also where tools and agents become less mystical.

A tool call is a loop inside the loop. The model writes structured text that means “call this function with these arguments.” Product code validates it, runs the outside thing, receives a result, and places that result back into context. If the call is malformed, unauthorized, too slow, or surprising, the product must decide whether to retry, ask the user, fall back, or stop.

An agent is not a different species of model. It is a product loop wrapped around a model: plan, call a tool, read the result, update context, continue. A multi-agent system splits that loop into roles, but now it must manage shared memory, duplicate work, disagreement, and runaway loops.

The useful test is plain:

Did the extra loop improve the task enough to pay for the extra time, cost, and failure surface?

Images follow the same pattern. An image is cut into patches or otherwise converted into model-readable number tables. A vision-language model can combine those image-derived representations with text. The exact design varies, but the journey is familiar: human content becomes numbered pieces or number tables, then the model operates on those numbers.

The prompt now exists.

It is still text.

And the model does not read text.

Text Becomes Little Numbers

The model reads integers.

Before the prompt can enter the model, a tokenizer cuts text into pieces and maps each piece to a number. A token might be a word, part of a word, punctuation, whitespace, an emoji fragment, or a byte-level piece of text. A token id is the integer assigned to that piece.

Try changing the text below. The exact ids are toy ids, but the shape is real: text becomes pieces, pieces become numbers.

Why not just use words?

Because human text is messy. Names appear. Code appears. Typos appear. New slang appears. Many languages mix scripts. A fixed word dictionary would constantly fail. Subword tokenization is the compromise: common words can be single tokens, while rare words are built from smaller pieces.

After tokenization, the prompt looks conceptually like this:

[128000, 9125, 374, 279, 2768, 315, ...]

Those ids do not contain meaning by themselves. They are more like library card numbers. Token id 9125 tells the model which row to pull from a learned table.

That first pull is an embedding lookup. The row it pulls is a vector: a list of numbers. Put one vector row beside every token position, and the prompt stops being prose. It becomes a table.

This is the first clean crossing:

Human text has become a rectangular block of numbers.

The tokenizer also explains many surprises. A short-looking code identifier can become many tokens. A compact log line can become expensive. A prompt injection hidden in retrieved text can consume enough token budget to push useful evidence out.

Tokens are not a footnote. They are the accounting unit for cost, context, waiting time, and the later memory the model must keep while generating.

Short Text Can Be Long to a Model

Humans count words.

LLM services count tokens.

That difference creates small surprises. A normal English sentence may tokenize compactly. A stack trace may explode. A long code identifier may split into many pieces. Random-looking base64, minified JSON, logs, mixed scripts, or pasted tables can look short on screen and still become expensive inside the model.

This matters before any clever reasoning begins.

Each extra token is another embedding lookup. During prefill, it is another row that must pass through the transformer. During attention, it may be another position to compare with earlier positions. During generation, it may become another key/value entry remembered by the cache.

So tokenization is upstream of almost every cost:

The tokenizer is also a compatibility boundary. A model trained with one tokenizer cannot simply be served with another. The ids would point at the wrong learned rows. The arithmetic might still run, but it would be reading the wrong shelves.

So our rainbow question is no longer a sentence. It is a row of shelf numbers.

But the shelves only matter because someone filled them earlier.

Now we have rows of numbers.

But rows of numbers cannot answer anything unless the learned numbers are already waiting somewhere.

The Learned Numbers Were Already Waiting

The most expensive part of your answer happened before you arrived.

Months earlier, a training run took huge piles of text and played one game again and again:

Given the pieces so far, what piece should come next?

Start with one sentence:

Paris is the capital of France.

The model sees “Paris is the capital of” and guesses the next token. If it gives “France” a low score, the training program measures that miss with a number called loss. Then it works backward to discover which learned numbers should move.

A weight is one learned number. A gradient is a direction signal for a weight: move this number up a little, or down a little, if you want the loss to shrink. Backpropagation is the bookkeeping walk that carries blame from the wrong final score back through the operations that contributed to it. An optimizer turns those directions into careful nudges.

One sentence barely matters. Training is that tiny correction repeated across enormous mixtures of text and enormous numbers of weights. The point is not to memorize one line. The point is to make many shared numbers better at the next-token game across many contexts.

The first trained model is usually not yet a good assistant. It may continue text well, but a chat product wants more: answer questions, follow instructions, refuse harmful requests, use tools, cite sources when asked. So a second family of training steps shapes behavior after base training.

In post-training, examples or preferences become a signal. The signal becomes a loss. The loss becomes another weight update. The updated snapshot then has to pass evaluations before it can serve users.

The names can stay plain:

Different labs use different recipes. The steady rule is simpler: when you press Enter, you are not training the model. You are asking a frozen snapshot to run forward using your current prompt.

Your chat may produce logs or feedback for future training. But the answer you are watching comes from weights already loaded for this request.

The Snapshot Carries Old Scars

Frozen does not mean perfect.

It means the mistakes are frozen too.

If training data over-represented a bad pattern, the weights may carry that habit. If duplicated text leaked into the mixture, the model may repeat memorized fragments more often. If post-training rewarded confident answers too strongly, the model may learn to sound certain when the evidence is weak. If safety examples were too broad, it may refuse harmless requests. If they were too narrow, it may under-refuse dangerous ones.

Serving cannot repair those weights on the fly. It can add context, use tools, choose safer decoding rules, or route to a different snapshot. But it cannot make the frozen weights know a fact they never learned and never receive.

This is why hallucinations have to be diagnosed by layer.

Some are training failures: the learned habit is bad.

Some are prompt failures: the needed evidence was not in context.

Some are retrieval failures: the wrong chunk was supplied.

Some are decoding failures: the right evidence was visible, but the next-token path drifted.

Some are product-loop failures: a tool result was ignored, a citation was trusted too easily, or a policy gate fired at the wrong time.

The symptom is one sentence on your screen. The owner may be anywhere in the journey.

The Second Training Can Also Hurt

Post-training makes a base model behave more like an assistant.

But every shaping step has a cost.

A domain fine-tune can make a model better at one domain and worse outside it. The human version would be practicing only legal contracts for weeks and then becoming clumsy at casual email. The model version is called catastrophic forgetting: extra training improves one behavior while damaging behavior the base model already had.

Teams fight this by mixing old and new examples, making smaller updates, freezing parts of the model, and testing old skills before release.

Another tool is distillation.

In distillation, a smaller or specialized model learns from answers produced by a stronger model. The important shape is teacher and student. Instead of learning only from raw human-written text, the student learns from model-generated targets that carry some of the teacher’s behavior.

There are compact adaptation methods too. LoRA trains small extra matrices rather than rewriting every weight. QLoRA combines that idea with quantized base weights so adaptation can fit on smaller hardware. Those details matter when teams want a model to learn a style, task, or domain without paying the cost of a full training run.

Names multiply quickly: SFT, RLHF, DPO, GRPO, LoRA, QLoRA, distillation.

The clock-post version is simpler:

What signal changes future answers, and which learned numbers are allowed to move?

But a snapshot on disk is not yet a running model.

Files have to become a live machine.

A Model File Is Not a Model Yet

Before your request arrives, some worker has to load a model artifact.

An artifact is the bundle needed for serving: weights, tokenizer rules, chat format, safety settings, number formats, split plan, and configuration files. If those pieces disagree, the math can run while the meaning is broken.

The live copy is called a replica. A replica is not another mind. It is one loaded running copy of the same snapshot, ready to accept work.

Loading a large replica is not like starting a small web server. The worker checks files, verifies checksums, reads weight shards, copies bytes into CPU memory, moves them into GPU memory, warms GPU programs, allocates workspace, reserves room for future tokens, and only then announces that it can serve.

A shard is one stored piece of a larger model. A tensor is an organized block of numbers with a shape. The shape matters because GPU programs expect exact dimensions. If a worker loads the wrong block, the next operation may not know how to multiply it.

Think of a stage set arriving in crates. One crate has the wall. One has the lights. One has the floor. Opening a crate does not give you a play. The crew has to assemble the set, wire the lights, test the rig, and only then let the audience in.

That is model warmup.

The Weights Are Only the First Rent

Now comes the second surprise.

“The weights fit” is not the same as “the service works.”

A GPU must hold its share of weights, but also temporary workspace, communication buffers, allocator gaps, safety margin, and the growing memory for active conversations. The model file is the first rent. Every open stream keeps paying.

If one GPU in a group runs out of room, the whole running copy is in trouble. A deployment plan is therefore not just “70B on four GPUs.” It is a budget for what every GPU must hold at the instant the next decode step launches.

The Contract Can Break Before Math Starts

Several failures at this layer look like “the model is down” even though no model math has run.

The tokenizer file can mismatch the embedding table. The chat template can wrap messages differently from the format used in post-training. The split plan can expect four GPUs while the runtime has eight. A shard can fail checksum. A runtime can load the weights but not support the requested smaller-number format. A cold replica can be healthy but not yet warm, so routing to it creates a long pause before the first token.

Some failures are worse because they do not crash.

If the tokenizer and embedding table disagree, token id 128006 may point at the wrong row. The matrix multiply can still run. The output can still look like language. But the meaning is poisoned at the boundary.

So a production artifact is not “a file with intelligence inside.”

It is a contract:

If the contract is wrong, the first visible symptom may be a timeout, a refusal spike, nonsense output, or just “the new model feels worse.”

Now the model is warm.

Your prompt can finally enter the stack.

The Model Reads the Whole Prompt Once

The prompt is now a table: one row per token position, many numbers per row.

A transformer layer edits that table. It takes in one row per token and returns one row per token. The shape stays the same, but the numbers inside the rows change.

Stack many layers and each token row becomes more informed by its context.

The famous step is attention.

Take the phrase:

river bank
central bank

The word “bank” needs earlier words to make sense. Near “river,” it should lean toward land beside water. Near “central,” it should lean toward finance.

Attention is the mechanism that lets one token position look back at earlier positions and ask: which of you matters to me right now?

Each position creates three learned views of itself:

The query compares with keys. The comparison gives strengths. The strengths mix values. The mixed result flows back into the current token row.

If that sounds abstract, keep the library picture. The query is what you search for. The keys are labels on cards. The values are the records you actually read.

After attention, another part of the layer edits each position privately. This is often called an MLP or feed-forward network. If attention lets positions talk to each other, the MLP lets each position digest what it has received.

The layer does not replace the stream. It edits it and adds the edit back. Those add-back paths are residual connections. They help many layers accumulate useful changes without every layer having to rebuild the whole representation from scratch.

At the final layer, the model turns the last token’s row into one raw score for every possible next token. These raw scores are logits.

The model still has not chosen a word.

It has created a score landscape.

The Symbols Are Just the Same Story Tighter

You do not need the symbols to keep reading.

But the symbols are useful because they show where the pain comes from.

Let the hidden width be H, the number of token positions be T, and the number of attention heads be A. The layer starts with something shaped like this:

X: [batch, T, H]

The model multiplies X by learned tables to make queries, keys, and values:

Q = X Wq
K = X Wk
V = X Wv

Then it reshapes them so several attention heads can run side by side:

[batch, A, T, head_dim]

Each head compares positions with positions. That makes a T x T score table before optimized kernels avoid materializing the giant table in the naive way.

Double the prompt length, and the simple score table wants four times as many cells.

That is why long context is not just “more words.” It changes the size of the comparison problem.

The compact attention story is:

compare queries with keys
block future positions
turn scores into mix weights
mix the values

The causal mask is the “block future positions” rule. While generating token 42, the model may use tokens 1 through 41. It may not use token 43, because token 43 does not exist yet.

The Private Editor Matters Too

Attention gets the fame.

The MLP does a lot of the work.

If attention is how positions talk to each other, the MLP is how each position changes itself after hearing the room. It applies learned table operations to each token row independently. A gate can let some intermediate features through strongly and damp others.

That private edit is computationally large. In many transformer layers, the MLP uses big matrix multiplies that dominate arithmetic work. It is less intuitive than attention because it does not have the friendly “this word looks at that word” story. But it is where a lot of per-token transformation happens.

Some models replace one big MLP with a mixture of experts. Picture a workshop with many specialist benches. For each token, a small router chooses a few benches. Only those experts run, so the model can contain many weights without using all of them on every token.

That saves compute per token, but creates a new systems problem. Tokens have to be sent to the GPUs that hold their chosen experts. If too many tokens choose the same expert, that expert becomes a bottleneck. If experts live across machines, the model has created an all-to-all communication problem inside the layer.

Again the pattern repeats:

A clever model trick becomes a scheduling and communication trick when it serves real traffic.

The prompt has now been read once.

But the answer is not one token long.

If the model had to reread the whole past from scratch every time, generation would crawl.

So it remembers the useful parts.

The Past Becomes a Cache

Imagine a 500-token answer.

For the first answer token, the model reads your whole prompt. For the second, it needs the prompt plus the first answer token. For the third, it needs the prompt plus the first two answer tokens.

The earlier work does not change.

So serving systems keep a KV cache. KV means key/value. The plain idea is simpler: while reading earlier tokens, each layer creates reusable notes. During generation, the model can look back at those notes instead of recomputing the whole past.

The first phase is prefill. That is the big read of the whole prompt. It fills the cache.

The second phase is decode. That is the one-token loop. Generate one token, add its new notes to the cache, then use the enlarged cache to generate the next token.

This is why two speed numbers matter:

The cache saves work, but it spends memory. Every active conversation carries remembered state until it finishes or is cancelled. Long prompts, long answers, and many simultaneous users all push on the same memory budget.

If KV memory fills, the problem starts to look less like “AI” and more like operating systems. The server may queue, reject, swap, evict, recompute, or route elsewhere. You can have arithmetic capacity left and still be blocked by memory.

The First Read and the Later Ticks Want Different Machines

Prefill and decode are not the same kind of work.

Prefill reads the whole prompt. It likes big chunks of work and fills the cache for many positions at once.

Decode advances one token at a time. It likes steady, low-latency turns. Each turn is smaller, but it must happen again and again, and token 257 cannot exist until token 256 has been chosen.

Some serving designs separate those phases. One pool specializes in prompt reading. Another specializes in decode. That can improve hardware use, but it creates a new problem:

How does the remembered state get from the prefill world to the decode world?

The KV cache may need to move, be shared, be rebuilt, compressed, restored, or spilled. What looked like “saved work” becomes data with a location, lifetime, owner, and transfer cost.

This is why long-context serving keeps producing new runtime designs. The difficult object is not only the model weight file. It is the active conversation state.

The Cache Has a Lifetime

A cache entry is not just “memory used by the model.”

It has an owner.

It belongs to a sequence. It lives on a device or in a memory block. It may be paged, reused, evicted, swapped, compressed, or freed. If the user cancels, it should die. If the stream finishes, it should die. If the request is preempted, the system has to decide whether to keep it, move it, or recompute it later.

This is why KV-cache allocators look a little like operating systems. They manage blocks. They fight fragmentation. They track which sequence owns which pages. They try to keep enough free space for the next request without wasting too much memory on gaps.

There is also a positive trick: prefix caching.

If many requests begin with the same prefix, the service may avoid rereading that shared beginning. System prompts, tool definitions, or repeated application scaffolding can create common prefixes. Save that work once, and later requests can start from the cached state.

But prefix caching has its own rules. The prefix must match exactly enough. The cache must still be valid for the same model, tokenizer, and prompt format. A tiny hidden instruction change can ruin the match.

The object keeps getting more concrete:

The past is not an idea. It is blocks of numbers with addresses and owners.

The cache explains the memory pressure.

Now look lower. Those blocks still have to be read by real hardware.

Where are those numbers physically moving?

The Numbers Touch Metal

Inside the GPU, the model is no longer language.

It is number tables moving between kinds of memory and arithmetic circuits.

Some time is spent doing math. GPUs contain many arithmetic units, including tensor cores, circuits built to multiply small number tables quickly.

But a number cannot be multiplied until it arrives.

That is the quieter half of the story.

GPU HBM is high-bandwidth memory packaged close to the chip. It can hold large streams of numbers. SRAM sits on the chip itself. It is faster, but much smaller. Registers are smaller still, and closer to the arithmetic.

Far and large. Close and tiny.

An LLM kernel is often a choreography for moving the right numbers from the large place into the fast place just before the arithmetic needs them.

Trace one attention tile. A tile is a small block of a larger table. A block of GPU threads pulls a slice of cached keys from HBM into faster on-chip storage. Tensor cores multiply query tiles by key tiles. The kernel keeps a running summary, streams the matching value tile, mixes it into the output, and moves on.

If the tile is chosen well, the chip reuses nearby numbers and avoids writing a giant temporary table back to HBM. If the tile is chosen poorly, the chip spends its time fetching and storing instead of multiplying.

This is why the same attention math can be slow or fast depending on the memory path.

The physical bottom is not “electricity” in the abstract. It is charge stored in memory cells, signals crossing package traces, transistor gates switching, SRAM banks feeding arithmetic units, network cards moving packets, and cooling systems carrying heat away so the chips can keep their clocks.

The model math and the data center have become the same story.

Smaller Numbers Change the Path

A weight stored in 16 bits takes twice as much memory as a weight stored in 8 bits.

That makes quantization tempting.

Quantization is the craft of storing numbers with fewer bits while trying to preserve behavior. It is like rounding measurements on a map. Round gently and you save little. Round aggressively and roads no longer line up.

There are several places to round.

Weight-only quantization shrinks the fixed model weights. That helps fit and bandwidth. Quantizing intermediate activations can speed or shrink temporary work, but mistakes there can be more visible because those numbers are being actively transformed. KV-cache quantization attacks the memory that grows with context and concurrency.

The important point is that quantization is not only a file-size trick.

It changes which kernels run. It changes memory bandwidth. It changes cache layout. It can change quality failures. A 4-bit model that fits may still be worse than an 8-bit model that barely misses the target. A quantized KV cache may save active memory while introducing subtle degradation on long contexts.

The practical question is not “can the model be smaller?”

It is:

Which numbers can be made smaller without breaking the behavior users care about?

One GPU may not be enough.

So the model is split.

And splitting creates a new problem: the pieces have to talk.

More GPUs Means More Waiting Lines

If a model is too large for one GPU, the service has a few levers.

It can store numbers with fewer bits. That is quantization: a careful rounding of model numbers to save memory while trying to preserve behavior.

It can split one big matrix operation across GPUs. That is tensor parallelism.

It can put different layers on different GPUs. That is pipeline parallelism.

It can run more full copies so different requests go to different replicas. That is data parallelism.

The trap is that “use more GPUs” is not a complete answer.

More GPUs add memory and arithmetic, but they also add communication. If GPU 0 computes one slice of a result and GPU 1 computes another slice, the next operation cannot pretend either slice is complete.

The chips need a coordinated exchange.

A collective operation is a group communication step. In an all-reduce, every GPU contributes partial values and every GPU receives the combined result. In an all-gather, every GPU contributes a shard and every GPU receives the assembled tensor.

That collective is a waiting point. The fastest GPU waits for the slowest one.

Now the interconnect is part of the model. NVLink, NVSwitch, InfiniBand, or high-speed Ethernet are not just infrastructure around the answer. They can be inside the path of each generated token.

Parallelism trades one pain for another: less pressure on local memory, more pressure on communication and synchronization.

The model has now produced raw scores.

The answer still has not been written.

It has to choose one next piece.

The Raffle for the Next Piece

At the end of the stack, the model has one raw score for every token in its vocabulary.

The simplest rule is greedy decoding: pick the highest-scoring token.

That is easy, but it can make writing brittle or repetitive. Chat systems often sample instead.

Sampling means choosing randomly, but not equally. Imagine a raffle. Better tokens get more tickets. After “Paris is the capital of,” the token for “France” gets many tickets. “Banana” gets almost none.

The raw scores are not ticket counts yet. They can be negative, huge, or tiny. They do not add up to anything useful. A function called softmax turns raw scores into probabilities: ticket shares that add up to 1.

Then the product can reshape the raffle:

Then one token is drawn.

That token is appended to the sequence. The KV cache gets one new set of notes. The model runs again. Another raffle. Another token. Another stream event.

That last paragraph is the loop.

One visible token is not one operation. It is a scheduler decision, a batch slot, dozens of layer passes, memory reads, matrix-multiply tiles, possible GPU-to-GPU exchanges, a logits vector, a sampling choice, a cache append, a detokenized fragment, and a network flush.

Reasoning models add another wrinkle. They may spend extra serving-time compute before the final answer: hidden scratch tokens, multiple sampled attempts, verifier passes, code execution, search, or tool calls. The product may show none of that internal work and stream only the final answer.

So “reasoning” in the chat app often means more one-token loops, more verification, more queue occupancy, and more memory pressure in the machinery underneath.

The Small Model Can Run Ahead

There is a speed trick that sounds like cheating.

Let a small model guess the next few tokens.

Then let the large model check them.

This is speculative decoding. The small draft model proposes a short future. The large target model verifies several proposed tokens in parallel. If the draft was right, the system accepts multiple tokens for roughly one expensive target pass. If the draft was wrong, the system falls back without changing the target model’s intended probability distribution, depending on the exact algorithm.

The user sees faster text.

The system sees a bet.

If the draft model is often right, the bet pays. If it is often wrong, the checking overhead may not help. The trick works best when the small model is cheap enough, aligned enough with the large model, and the serving stack can schedule the draft and verify work without creating new bottlenecks.

This is the same theme again: the math idea is only half the story. The serving win depends on the queue, memory, kernels, and batch shape around it.

Reasoning Spends Invisible Tokens

A normal chat request is already variable length.

A reasoning request is variable length twice.

The final answer length varies. The hidden or auxiliary work varies too.

The extra work might be hidden scratch tokens, visible reasoning-like text, multiple sampled solution attempts, verifier passes, code execution, search, or tool calls. The product may show only the final answer, but the scheduler still has to reserve room for the work that produced it.

That changes the front door, the cache, and the queue.

The front door must estimate a larger promise. The cache may hold hidden state the user never sees. The scheduler may have to protect ordinary streams from a few long reasoning jobs. The stream gate may hold the final answer until checks finish.

“Think harder” is not free.

At the bottom, it means more iterations, more memory lifetime, more tool time, and more ways for a request to be interrupted.

The next piece exists now.

But it is still an id.

It has to become text without breaking the stream.

The Stream Comes Back Alive

The serving worker detokenizes the chosen token id back into bytes or text fragments. The service frames the fragment as a stream event. The edge forwards it over the open connection. The browser decodes bytes, updates UI state, and paints.

That is why you can see half a sentence.

The server is not waiting for the whole answer. It is releasing pieces as soon as product rules allow.

But release is not always “print immediately.” Some tokens are word fragments. Some byte fragments do not form a valid UTF-8 character until the next piece arrives. A JSON tool call may be nonsense until its braces close. A citation marker may need source metadata. A safety check may hold a risky phrase until neighboring text makes it clear.

Streaming is a small protocol inside the product: buffer when needed, flush when safe, and keep enough state to stop cleanly.

Cancellation is the reverse journey. You hit stop. The browser closes or aborts. The edge marks the stream dead. The serving frontend marks the sequence cancelled. The scheduler removes it from future iterations. The KV allocator releases its blocks.

If a GPU kernel is already running, that iteration may finish. But the next one should not include work nobody will read.

The one-user loop is complete.

But the service is not serving one user.

Everyone Else Is in the Loop Too

Your request is one colored line in a giant moving map.

One user wants low latency: little waiting for their own answer. The provider wants high throughput: many useful tokens served per second across everyone.

Those goals fight.

Think of a bus. Leaving immediately is great for the first passenger, but wastes seats. Waiting until the bus is full uses the bus well, but makes the first passenger wait. A GPU has the same tension.

For one request, decode generates one token at a time. That is too little work to keep a large GPU busy. The trick is to generate the next token for many requests in the same model step.

People do not arrive in neat groups. One prompt is short. One is huge. One answer ends after ten tokens. Another keeps going for a thousand. A fixed batch would quickly fill with holes.

So modern serving systems rebuild the batch while work is already moving. That is continuous batching, also called in-flight batching.

Every active conversation has a record attached to it: input length, generated length, prefill or decode state, KV-cache pages, remaining token budget, priority, and cancellation status.

The scheduler repeatedly asks:

Which records can fit in the next model step without breaking compute and memory budgets?

Now cancellation is not a UI nicety. It frees cache pages. A huge prompt is not just slow. It may need to be read in chunks so dozens of active streams do not freeze. A priority decode is not merely another queue item. It is a visible next token waiting on this iteration.

The Batch Is Rebuilt Every Tick

Ordinary web servers often think in whole requests.

LLM serving cannot stop there.

A generated answer is many small iterations. On each iteration, the scheduler chooses which active conversations advance by one step. A request that already finished should leave. A newly arrived request may enter. A cancelled request should disappear. A long prompt may need prefill time. A short decode may need one quick next-token turn.

This is continuous batching.

The batch is not a sealed box. It is a moving roster.

That moving roster creates real policy choices. Should a huge prompt be allowed to block many streams while it is prefilling? Should short requests jump ahead? Should a paid tier get smoother inter-token latency? Should a nearly finished answer get priority so it frees memory sooner? Should a request with a giant future output limit be admitted now or held outside?

None of these are purely mathematical questions. They are product behavior leaking out of scheduling policy.

The best scheduler is not the one that maximizes one number. It protects useful work:

The user sees typing.

The runtime sees a live packing problem.

Under load, failure can be sudden. Queues rise. Users wait. Some cancel or retry. Retries add traffic. More open streams hold more cache. More cache pressure creates more queueing.

So the next question is not “can one rainbow answer finish?”

It is: can thousands of unfinished answers keep moving without knocking each other over?

The Fleet Keeps the Trick Boring

A good LLM service makes the miracle feel boring.

You press Enter. Text appears.

Behind that calm surface, the fleet is always changing. Some replicas are warming. Some are being drained for an update. Some hold old model snapshots. Some hold a canary snapshot. Some are overloaded because a feature just sent a traffic spike. Some are slow because one GPU link is unhappy.

This is why the fleet needs a diagnosis habit. The same sentence from a user, “the model feels worse today,” is not a bug report with one obvious owner.

It might mean the new snapshot changed behavior. It might mean a retrieval index missed evidence. It might mean a safety gate got stricter. It might mean one GPU in a tensor-parallel group is slow, so tokens arrive unevenly. It might mean the frontend is buffering stream fragments badly. The human symptom is vague because many layers meet at the same little piece of text.

So the fleet’s job is not only to serve. It is to notice which loop is drifting before the user has to name it.

A new model rollout is not just a behavior experiment. It is also a systems experiment. A safer model may be slower. A faster runtime may change rare formatting behavior. A smaller-number build may fit more traffic but damage quality in edge cases.

A Rollout Is Two Experiments

A new snapshot can be tested without showing it to users.

That is shadow traffic. The old snapshot answers the user. The new snapshot receives a duplicate prompt in the background. Its answer is logged, scored, compared, and discarded. Shadow traffic lets a team see whether the new model is slower, costlier, more likely to refuse, worse at citations, or better at a target task before users depend on it.

If the shadow path looks healthy, a canary begins.

Now a small visible slice of users reaches the new snapshot. The provider watches two families of numbers at once.

Behavior numbers:

• did users regenerate more?
• did citation faithfulness improve or fall?
• did safety violations change?
• did the model follow tool formats?
• did human or model graders prefer it?

Systems numbers:

• did first-token time change?
• did inter-token latency change?
• did KV-cache pressure change?
• did GPU utilization change?
• did error rate or cancellation rate move?

A rollout fails if either family fails. A model can be smarter and too slow. A runtime can be faster and subtly worse. A safety change can reduce bad output and over-refuse good requests.

This is why rollback is not only for crashes. Sometimes rollback means: the service is healthy, but the answers got worse.

The same complaint can come from different layers:

Stability is a set of feedback loops. Watch queue depth. Watch first-token time. Watch inter-token time. Watch KV occupancy. Watch hardware health. Watch quality scores by snapshot. Watch citation faithfulness. Watch policy violations. Then decide what harm to accept.

Load shedding is deliberate refusal. A clear failure for some users can be safer than letting everyone enter a queue so long that they time out and retry.

Autoscaling helps later, but not instantly. A warm LLM replica may require copying hundreds of gigabytes of weights, verifying shards, warming kernels, allocating cache blocks, and gradually routing traffic. During the overloaded minute, the fast controls are admission, routing, batching policy, output limits, and rejection.

Stability Means Choosing Which Pain

Every control protects something and harms something.

Rejecting long prompts protects KV cache, but hurts users with legitimate long documents.

Lowering max output protects decode capacity, but may truncate good answers.

Routing to a smaller model protects availability, but may reduce quality.

Tightening a safety check reduces bad output, but may over-refuse.

Holding output for a stronger check improves safety, but makes the stream feel slower.

So stability is not one switch. It is a set of named tradeoffs. A mature service names both sides:

The discipline is to map the symptom to the loop that owns it. A late first token may be admission, prefill, routing, or cold replicas. Bad citations may be retrieval, prompt packing, model behavior, or evaluation coverage. Slow streamed tokens may be GPU kernels, collectives, queue policy, or KV pressure.

The journey matters because it narrows the fix.

The fleet’s job is not to make one answer possible once.

It is to make the next answer possible again and again, while traffic, models, prompts, memory, hardware, and quality all keep moving.

The Loop Closes

Now go back to the first visible word.

It did not come from “the model” in one vague sense.

It came from a loop:

Your browser opened a stream. The front door admitted future work. The app packed a prompt larger than the sentence you typed. The tokenizer cut it into numbered pieces. A warm replica held frozen learned numbers. Transformer layers edited rows of numbers. The KV cache remembered the past. GPU memory fed arithmetic circuits. Distributed workers may have exchanged partial results. The sampler chose one next id. The stream gate released safe text. The browser painted it.

Then the chosen piece was fed back in.

And the loop ran again.

That is why the answer appears before it is finished. There is no finished answer yet. There is only the next piece, chosen under the pressure of everything before it.

LLM Chat at Every Level

Further Reading

• Vaswani et al., Attention Is All You Need (2017). The transformer paper.
• Gao et al., The Pile (2020), and Lee et al., Deduplicating Training Data Makes Language Models Better (2021). Data mixtures, filtering, repetition, and memorization effects.
• Brown et al., Language Models are Few-Shot Learners (2020). GPT-style autoregressive scaling and prompting.
• Ouyang et al., Training language models to follow instructions with human feedback (2022). A public instruction-following and preference-training pipeline.
• Hoffmann et al., Training Compute-Optimal Large Language Models (2022). The Chinchilla scaling-law result.
• Rafailov et al., Direct Preference Optimization (2023), Hu et al., LoRA (2021), and Dettmers et al., QLoRA (2023). Preference training and compact adaptation.
• Lewis et al., Retrieval-Augmented Generation (2020). Retrieval plus generation.
• Yu et al., ORCA (2022). Iteration-level scheduling for generative serving.
• Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (2023). KV-cache paging and serving throughput.
• Zhong et al., P/D-Serve (2024), vLLM Disaggregated Prefilling, CacheFlow, and Tutti. Prefill/decode separation and KV cache as movable state.
• Dao et al., FlashAttention (2022), and Shah et al., FlashAttention-3 (2024). Attention as memory-traffic engineering.
• Shoeybi et al., Megatron-LM (2019), and Rajbhandari et al., ZeRO (2019). Model parallelism and memory sharding.
• Frantar et al., GPTQ (2022), and Lin et al., AWQ (2023). Public post-training quantization methods.
• NVIDIA CUDA C++ Programming Guide, NCCL collectives, and TensorRT-LLM. The public hardware and runtime references behind the serving path.