We taught a machine to write Thirukkural. Here is everything that happened.
A small AI that has read the 1,330 couplets and nothing else — writing a new one, live, one letter at a time. Same design as ChatGPT, a million times smaller. Every picture today is the real machine.
live · a couplet that did not exist a second agorunning in your browser
This machine has 164,160 adjustable numbers inside it and learned from 82,954 letters of text. The models behind ChatGPT have hundreds of billions of numbers and learned from most of the internet. Same design, different size. What you read above is Tamil that looks and sounds like a kural and usually means nothing — exactly what a machine this small can do. The panel under it says what can honestly be said: which words are real, and which real kural it is closest to. The chapters explain why, and what size changes.
→ next · ← back · F fullscreen · O overview · S speaker notes · D one level deeper
01 · the one trick
Every AI that writes does exactly one thing: guess what comes next.
Your phone does it when it suggests the next word. Show the machine some text, it guesses the next letter, adds it, and guesses again. ChatGPT, Gemini and this machine run the same loop; only the quality of the guess differs.
game · you vs the machine, one letter
how sure the machine was about each option
Pick a letter above. The machine's guess stays hidden until you commit.
The bars are not pre-written. Each time you press a button, the page runs the whole machine on the visible text and reads off how much it believes in each of the 46 letters it knows.
02 · the only book
Everything the machine will ever know is in this one box.
A child who has read exactly one book. 1,330 couplets, 133 chapters, three parts: virtue, wealth, love. Four words, three words, a full stop. No meanings, no titles, no translations were shown to it.
1,330
couplets · 133 chapters
82,954
letters in total
45
different letters
4 + 3
words per line, in 1,305 of 1,330
the book · all 1,330 · click a couplet for its meaning
Source: the public-domain text of Thirukkural; English rendering by Yogi Suddhananda Bharati; meanings from Mu. Varadarajan's Tamil urai and a prose English explanation; the transliteration is generated on this page and is approximate. The translations and meanings are shown for you — the machine never sees them. It sees only the 82,954 letters of the couplets.
03 · letters → numbers
Step one: give every letter a number, like a locker in a school corridor.
Line up every different letter, number them 0 to 45. From here on the machine never sees a letter, only its locker number. Space and new-line get lockers too.
the locker list · one number per letter
All 46 lockers. A letter the book never contains has no locker, so the machine cannot even see it — try typing an English letter into the Tamil version and it vanishes.
Space (locker 1) and new-line (locker 0) are letters too, as far as the machine is concerned. It learns where lines end the same way it learns spelling — which is how it learns that a kural has two lines.
03 · letters → numbers
Why the big models use chunks, not letters.
Glue the two most common neighbouring letters into one new token; repeat. Syllables assemble themselves. GPT-4 has ~100,000 chunks learned mostly from English — so Tamil costs two to four times more tokens. The token tax.
the shorthand · learning chunks from the book
the same line, in chunks
latest merges (most common pair first)
This is called byte-pair encoding. GPT-4's vocabulary has ~100,000 such chunks, learned mostly from English, so an English word is often one chunk while a Tamil word shatters into three to six. That is the token tax: the same sentence costs Indian languages two to four times more, in money and in memory. A vocabulary learned with Tamil in the mix would not have that problem.
04 · a map of letters
Step two: every letter gets a place on a map. Meaning is where you stand.
A spice rack: jars you use together end up next to each other. Each letter's place is written as 64 numbers. It starts random; after learning, letters that behave alike sit together. Nobody told it what a vowel is.
the map · 46 letters × 64 numbers, from the trained machine
click a row · the 64 numbers behind one letter
positivenegativeeach cell is one learned number
row: —
The map squashes 64 numbers down to 2 so you can see it ( of the pattern survives the squash); the neighbours list uses all 64. Press random → trained to watch order appear from noise. Nobody programmed the groups: vowel signs cluster, consonants cluster, and the full stop that ends a couplet sits on its own.
05 · the spotlight
Step three: every letter shines a spotlight back over the sentence and asks, “which of you matter to me?”
“The bank was steep and muddy” — you know which bank because you looked back. Every letter does that, all at once. Bright = matters. The white triangle: no looking forward; the future doesn't exist yet. This is attention.
the spotlight · from the trained machine, live
dimbrighthover a row · click to pin
hover a row on the map
Each cell is a real number the trained machine computes. Darker means the row letter paid more attention to that column letter. The machine has four spotlights per round and three rounds; try spotlight 2 and 3 — each one learns its own habit: one watches the previous letter, one the start of the word, one the line break.
05 · the spotlight
The arithmetic behind one spotlight, on the machine's own numbers.
Optional. Six steps: the letter, three small lists (query, key, value), scores, hide the future, turn scores into shares that add to 100%, blend.
Q · K · V walkthrough · round 1, spotlight 1
Each spotlight (“head”) works on a 16-number slice of the 64. Four heads run side by side and their results are joined back into 64 numbers, so one round can ask four different questions of the past at once.
06 · the machine
The whole machine is one small routine, repeated three times.
Spotlight between letters, then a small thinking step within each letter. Each part adds to what the letter already had — nothing is thrown away. Three rounds, then a scoreboard.
the design · what one letter passes through
one letter's 64 numbers · how they change through the three rounds
Each strip is the same letter's 64 numbers after each stage. Blue positive, pink negative. The last strip is what the scoreboard reads to guess the next letter. It began as map-row plus seat-row; everything else was added by the spotlights and the thinking steps, never overwritten.
06 · the machine
Where the 164,160 numbers live — and how they become 175 billion.
Every arrow is a table of adjustable numbers: dials. Ours has 164,160. The same four ingredients, with two of them turned up, is GPT-3.
dial counter
Rule of thumb: dials ≈ 12 × rounds × d² plus two vocabulary tables. GPT-3: 96 rounds, d = 12,288 → 175 billion. Stored at 2 bytes each, our machine is ~330 KB (it is embedded in this page) and GPT-3 is ~350 GB.
07 · the weighted die
Step four: score all 46 letters, then roll a weighted die.
The scores become percentages that add to 100. The machine does not take the top letter; it rolls a die weighted by those percentages. Temperature = how adventurous the die is. A hallucination is a die roll on a flat scoreboard.
the scoreboard · live
Watch the bars while it writes. When one bar is near 100% the machine is on rails (it is inside a word it has seen many times). When the bars are flat it is guessing — and with the temperature high it takes those guesses. An AI “hallucination” is not a bug in some separate part. It is a die roll on a flat scoreboard.
08 · learning
Start with every dial at random. Guess, get corrected, nudge. 3,000 times.
Like learning a song: sing it wrong, hear the right note, adjust a little. Surprise is the score to drive down. Backpropagation tells every dial which way to turn. Nobody writes a rule about Tamil; the rhythm and the full stop are whatever settings made surprise fall.
the education · surprise going down, and what it wrote along the way
surprise on the book it learns fromsurprise on couplets it never sawx: learning step, square-root scale
09 · teaching a habit
A machine that has only read continues. To make it respond, show it examples of responding.
The gap between GPT-3 and ChatGPT. Same loop, 1,500 more steps, on text shaped as request → response: a chapter title, then a couplet. One new symbol and a format — that is all a chat template is.
before vs after · same request, same die, live
after the first education only
after fine-tuning
The first machine has never seen a #. It reads the title as the first line of a couplet, writes a plausible second line, and stops — it can only continue the pattern it knows. The fine-tuned machine has learned that a title line is followed by a two-line couplet and a blank line — and, weakly, that the topic's words should follow. Weakly because 1,330 examples is tiny; ChatGPT's fine-tuning used tens of thousands of human-written conversations, on a machine that had read the internet.
10 · a judge, not a teacher
Let the machine try twice, say which is better, make that one more likely.
A teacher shows the answer; a judge only says “this one.” RLHF: people judge pairs; a second machine learns to imitate the judges; the writer is nudged toward what the judge prefers. You are the judge.
move 1 · you are the judge0 preferences logged
At scale this is a job: thousands of people, hundreds of thousands of comparisons, with written guidelines about honesty, harm and helpfulness. Those guidelines are where a company's values enter the machine. Nothing in the maths knows what “good” means; the judges do.
10 · a judge, not a teacher
The update, actually run — and what over-doing it looks like.
Our stand-in judge is a rule about shape (two lines, 4 + 3 words, full stop, not a copy). 600 requests, two attempts each, keep the pairs the judge can separate, nudge toward the winners while staying on a leash to the original. Measured:
move 3 · couplets that pass the judge, as the update runs
Measured on 300 fresh couplets at each checkpoint, same die settings throughout. The machine was never shown the rule; it was shown pairs of its own attempts and which one won.
the three machines, side by side · live, with the judge's verdict
judge: 2 lines · 4 + 3 words · ends with . · not a copy
first education only
+ fine-tuning
+ the judge
11 · watch it think
One letter, start to finish. Then again.
Left: the letters so far. Each column of lights: the newest letter's 64 numbers after a stage. Orange fan: that round's spotlight. Right: the scoreboard; the die's pick turns green and joins the text.
the machine, thinking · live
positive numbernegative numberspotlight (brighter = more attention)the letter the die pickedthe seven boxes below are the same pass, in words
1 · text so far
2 · locker numbers
3 · map + seat
4 · 3 rounds of spotlight + think
5 · scoreboard
6 · roll the die
7 · add it, go again
Stages 2–6 run once per letter: a 60-letter couplet is 60 full passes through the machine. ChatGPT shows its answer word by word for the same reason — you are watching this loop.
11 · watch it think
What size changes, and what it doesn't.
ingredient
this machine
GPT-3 class
changes the idea?
tokens
46 letters
~50k–100k chunks
no
numbers per token
64
12,288
no
rounds
3, with 4 spotlights
96, with 96 spotlights
no
reads at once
128 letters
100k–1M tokens now
no
dials
164,160
175,000,000,000
no — it is what emerges that changes
the book
83 thousand letters
300 billion words
this is the ceiling
education
11 minutes, a laptop
weeks, thousands of chips
no
judge
a rule about shape
hundreds of thousands of human judgements
no — the risk grows
Two things appear only at size: learning from the prompt itself and knowledge of the world. Our machine has neither, because 83 thousand letters contain neither.
12 · build your own
You can build this. Change the text file.
One text file and about 120 lines of code made everything you saw. Bharathiyar, Kabir, Shakespeare, film lyrics, your grandmother's recipes. A notebook in the course kit does it end to end: upload a file, press run, ten minutes.
step
what you do
chapter
1 · the book
One .txt, one example per paragraph, blank line between. 50,000+ letters.
2
2 · lockers
Automatic. Any script.
3
3 · education
Run the cell; stop when surprise on unseen examples stops falling.
4–8
4 · habit
Add a # label line per example; train a little more.
9
5 · judge
A pass/fail rule or human picks; pairs; the preference update.
10
6 · share
Export; the page's JavaScript runs any machine of this shape.
11–12
no programming neededlabs/build-your-own-poet.ipynb — open in Google Colab, upload a text file, Run all. The interactive page how-llms-work.html has every widget from today plus a “one level deeper” fold per chapter and the full code.