The model's last layer does not emit probabilities. It emits one raw score per token in the vocabulary — a logit. Logits are unbounded: +8.2, −1.4, 0.03. To turn a list of scores into a list of probabilities that sums to 1, you run softmax: exponentiate everything, then divide by the total.

Now look where T sits: it divides the logits before the exponential. That one placement explains the whole slider. T < 1 stretches the gaps between scores apart, so exp() exaggerates the leader — the peak sharpens. T > 1 squeezes the gaps together, so the field flattens. T → 0 makes the top logit infinitely dominant, which is why temperature 0 is just argmax: always take the highest-scoring token, same answer every time. Temperature isn't a mysterious creativity dial. It is one division.

Sampling is also why temperature 0 is not fully deterministic in production: floating-point addition on GPUs isn't associative, so batching can reorder sums and flip a near-tie. Close to reproducible, never guaranteed.

You just reduced an error by eye. A lab training for four months cannot do that — so what number do they actually watch? One: the loss. At every step the model predicts a distribution over the next token, and the true next token is known. Loss is how much probability the model put on the right answer, scored so that being confidently wrong hurts most:

Put 0.9 on the correct token and you lose 0.11. Put 0.1 on it and you lose 2.3. Put 0.001 on it and you lose 6.9. Confidently wrong is punished roughly twenty times harder than merely unsure — which is exactly the behaviour you want to train out.

Perplexity is that same number made human: exp(loss). Read it as "how many equally likely options is the model effectively choosing between?" Perplexity 1 means it always knows. Perplexity 50 means it's flailing among fifty. A modern model on ordinary English sits in the low single digits. That single falling curve, watched for months across thousands of GPUs, is the entire feedback signal of pretraining.

Worth knowing: loss keeps improving smoothly long after the model stops feeling smarter — and it says nothing about whether answers are true, safe, or useful. That gap is why Session 2 exists. Pretraining measures loss; engineers measure evals.

Embeddings — meaning as coordinates. Before anything else happens, each token is turned into a long list of numbers: thousands of dimensions, of which a picture can show two. Tokens used in similar contexts land near each other — idli sits beside dosa, and both are a continent away from GPU. Because meaning is now geometry, it supports arithmetic: the famous party trick is king − man + woman ≈ queen. These are learned rows of a table, not hand-written; direction carries the meaning, and the same idea becomes a search engine over your own notes in Session 4.

Attention — every token looks at every other. Coordinates alone cannot tell you what “it” refers to in “the trophy didn’t fit in the suitcase because it was too big”. So each token weighs every other token in the context and blends in what matters: “it” leans on trophy; change big to small and the same mechanism re-weights instantly toward suitcase. That mechanism is the T in GPT — the Transformer, from the 2017 paper “Attention Is All You Need”. Two honest footnotes: attention is order-blind (a set has no order), so position is added into each embedding first — positional encoding, today usually RoPE; and “every token looks at every other” costs n² — double the context, quadruple the work, which is why a 1M-token window was a headline and why Session 4 retrieves three paragraphs instead of pasting the textbook.

Want the live versions? The pre-class page how-llms-work.html has the embedding map and the attention heat-map running on a real model (chapters on embeddings and attention); Session 4 re-teaches embeddings with the Gemini embedding API and real cosine numbers.

The demo above has one slider, parameters. Reality has three, and they have to move together: parameters (how many knobs), data (how many tokens you train on), and compute (how many GPU-hours you can pay for). Around 2020 researchers found these trade off in a smooth, boringly predictable way — scaling laws. Loss falls as a power law in each one. That predictability is the real reason billions got invested: you could forecast the capability of a model that did not exist yet.

The correction that changed the industry came in 2022. Everyone had been building models that were too big and too under-fed. The Chinchilla result showed that for a fixed compute budget you should scale parameters and training tokens together — roughly 20 training tokens per parameter. A smaller model trained on far more data beat a much larger one trained on less. Overnight, "make it bigger" stopped being the strategy.

Keep the shape of the argument, not the constants. The exact ratio has been revised more than once and depends on how you count. What has held for six years: capability follows compute in a predictable curve, and the curve is smooth even when the abilities that pop out of it feel sudden. That's the honest version of "emergence."

Look at the naming across all three kitchens: Flash and Pro. Haiku and Opus. mini and full. Every lab sells the same two-tier menu, and the free tier you'll use today is always the small one. Three techniques make a small model punch far above its size — and they're the reason your lab is free.

Now the practical consequence, which is the whole point of knowing this: "which model?" is not a question about intelligence, it's a routing decision. The mature production pattern is hybrid — the small fast model handles the routine 90% of traffic, and you escalate only the hard 10% to the expensive one. You'll make exactly that trade with real numbers in Session 6.

Also why benchmark tables mislead: a Flash-class model and a Pro-class model can score within a point or two of each other on a public benchmark while differing 10–20× in price and several-fold in latency. The gap shows up on your hard cases, which is a thing only your own eval set can tell you.

Good objection — and the answer is the single most useful piece of plumbing to know. If the app really re-ran a 50,000-token conversation through the whole network on every turn, chat would be unusable. It doesn't, because of the KV cache.

Remember attention: each token computes a key and a value that later tokens look at. For text already in the context, those keys and values never change — token 4,001 does not alter what token 12 was. So the model computes them once and keeps them. Answering therefore splits into two very different phases:

Three real-world facts fall straight out of this, and you'll meet all three later in the course:

Long chats get slow and expensive because the cache grows. It costs GPU memory proportional to context length, and it's per-conversation — which is a large part of what you're paying for.

The cache dies when your request ends, so a fresh call re-reads everything from scratch. That's why providers sell context caching: pin a long unchanging prefix (your system prompt, a big document) on their side and pay roughly 10% to reuse it. Same idea, rented. It's a headline cost lever in Session 6.

Put the stable part of your prompt first. Prefix caching only works up to the first byte that differs, so a prompt built as [fixed instructions][fixed document][changing question] is dramatically cheaper than one that varies at the top. Prompt order is a cost decision, not just a style one.

So the honest version of the slide above is: it re-sends everything, but it doesn't re-read everything. The illusion of memory is real; the re-computation is not.