AI beyond text:
eyes, ears and a paintbrush
Vision that reads your mess bill, pictures conjured out of pure static, voices you can't tell from human — and all of it still the one loop you met this morning.
Still true after the break?
Same loop.
New kinds of tokens.
Session 1's machine never cared what tokens meant. So: chop an image into little squares, turn each into an embedding — the attention loop reads a photo the way it reads a sentence. Chop audio into ~20-ms slices — it reads sound.
"Multimodal" isn't a new brain. It's new eyes wired into the same brain.
How a model reads a picture
Patches are visual words
The image splits into a grid of small squares. Each patch → an embedding → the same attention loop from Session 1. "What's in this photo?" is literally next-token prediction with picture-tokens in the context.
This is why vision arrived so fast once transformers worked — the hard part was already built.
Take "patches are visual words" completely literally, because the billing does. The image is cut into a grid — classically 16×16 pixels per patch — and each patch is flattened and multiplied by one learned matrix into a vector the same width as a word embedding. From that point the network cannot tell picture from prose. This is the Vision Transformer (ViT, 2020), and its whole contribution was noticing that you didn't need a special architecture for images at all.
Two consequences you can feel in the lab:
Resolution costs tokens, quadratically
Double an image's width and height and you quadruple the patch count. Providers therefore tile large images and charge roughly per tile — a full-page scan can cost more than a page of text. Downscaling before upload is a real cost lever.
Small text falls off a cliff
If a digit is smaller than a patch, its evidence is averaged into a single vector alongside its neighbours. There is no zoom. Below a certain size the information is simply gone — which is why a low-res receipt fails suddenly rather than gradually.
Counting is structurally hard
Attention pools and summarises; it does not enumerate. Asking "how many chairs?" makes the model estimate from a blended impression — the same disease as long multiplication in Session 1, and it shows up again on slide 13.
Practical version of all three: photograph documents straight-on and close, crop to the region you care about, and prefer one tight image over one wide one. You are not being fussy — you are raising the pixels-per-patch on the thing you actually want read.
It doesn't just see. It reads.
Documents
Receipts, invoices, mark sheets, forms → clean JSON. The most-shipped, least-glamorous vision use-case on the planet.
Handwriting
Your lab-record pages, doctor scrawl (mostly), whiteboard photos → text.
Charts & screens
Reads graphs, tables, app screenshots — "what's wrong with this UI?" works.
KYC pipelines
ID extraction + face match + tamper checks — how fintechs onboard you in minutes. This is my day job.
Accessibility
Live alt-text for blind users — describing the visual web. Quietly life-changing.
Field work
Leaf-disease detection from a farmer's phone photo. No lab needed.
Will it read it? Place your bets
Reading pictures is prediction.
Generating them reverses noise.
Many image generators use a different learning loop: add noise to real images, train a network to predict what noise was added, then start from pure noise and repeatedly remove it to create a plausible image. Some newer systems generate image tokens autoregressively; the families now coexist.
It's called diffusion. Watch it happen →
(One 2025 twist: OpenAI's 4o images are generated token-by-token, autoregressively — the loop ate images too. Diffusion is still the concept to know.)
Diffusion: a picture emerges from static
Training: destroy, then learn to repair
Take billions of real images, add noise step by step until pure static, and train a network to reverse each step.
Generation: start from fresh random static and apply the repair over and over. The "repair" invents a plausible image — steered at every step by your text prompt ("a gopuram at sunset"), which is embedded with the same trick from Session 1.
Our canvas fakes the pixels, honestly — but the choreography is true: coarse structure often resolves before fine detail. Today's lab is vision-in only; image-generation availability and quotas vary, so treat the making side as optional.
Here is the twist that makes diffusion trainable. The network is not asked to produce a clean image — that would be an impossibly hard target. It is asked, given a noisy image and a step number: "which part of this is the noise?" Predicting noise is easy to supervise, because you added it, so you have the exact answer for free. Subtract a little of the predicted noise, and you have a slightly cleaner image. Repeat.
noisy = clean + noise forward: fixed recipe, no learning at all predict → noisê reverse: the only thing the network learns cleaner = noisy − ε·noisê repeat 20–50 timesThree knobs from that loop show up in every image tool you will ever touch:
Steps
How many repair passes. More steps, more coherence, more time and money. Returns flatten fast — the gap between 20 and 50 is far smaller than between 4 and 20.
Guidance scale
Run the prediction twice — once with your prompt, once without — and push away from the promptless one. Low: creative, loose. High: obedient, and eventually oversaturated and stiff. This is the "prompt strength" slider.
Seed
The starting static is random. Fix the seed and the same prompt reproduces the same image — the only reason image generation is debuggable at all.
And the reason this runs in seconds instead of hours: modern systems are latent diffusion. They don't denoise pixels. An encoder first compresses the image roughly 8× per side into a small latent grid, all the denoising happens there, and a decoder expands the result back to pixels at the very end — around 48× less to compute per step.
Same maths, different medium: video models denoise across space and time, which is why they cost so much more and why physics is where they still slip. And note the family tree — this is a different lineage from the next-token models in Session 1. Some newer image models are autoregressive after all; the two families now borrow from each other freely.
Image generation: magic with fine print
Details lie
The six-finger memes are fixed. The subtle stuff isn't: dense small text, exact logos, jewellery, background faces. Zoom in before you trust.
Style questions
Trained on human art. "In the style of a living artist" raises real ethical and legal debates — unresolved, ongoing.
Deepfake era
If images can be conjured, images stop being evidence. Provenance tools (watermarking, content credentials) are the counter-move.
The camera spent 200 years earning our trust. Diffusion ended it in two. Same lesson as Session 2: the output is plausible, not verified — now in pixels.
Speech: useful, but not uniformly solved
Speech → text
Often excellent, but accents, background noise, rare names, Tamil-English code-switching, and domain vocabulary still create silent errors. Lectures → notes is a useful workflow — not a solved problem.
Text → speech
Voices now genuinely hard to tell from human — tone, pauses, emotion.
Voice cloning
A few seconds of audio can clone a voice. "Your grandson" calling paati for an OTP is a live scam pattern in India right now — agree on a family password. Seriously. Tell your parents this weekend.
Video: the same diffusion family, plus time. Mid-2026 reality: native 4K clips with the soundtrack generated alongside the pixels (Sora 2, Veo 3.1, Kling 3.0). Physics still slips in complex scenes, and it costs real money — roughly $0.10–0.75 per second. Improving every quarter — what you're watching today is the worst it will ever be.
Your mother’s voice is no longer
proof of your mother.
Three seconds of audio clones a voice well enough to fool family — voice-clone scam calls already run at scale in India. The fix is not better ears; it is verification habits: a family code word beats a trained ear.
Disagree? Good. Bring it to the break — strongest counter-argument gets named on the closing slide.
All of it is one API call
The contents list mixes images and text freely. Audio files work the same way. Everything from Sessions 1–2 (prompting, format control, evals!) applies unchanged.
The production way: a response_schema constrains the JSON shape and reduces parser failures. It does not verify the extracted values; in the lab you'll compare both the structure and the evidence in the image.
In the lab you'll do this twice. First you'll ask for JSON in the prompt — and it works, until the day the model opens with "Certainly! Here's the data:" or wraps everything in a code fence, and your parser dies in production at 2 a.m. Then you'll pass a schema with the request, and it stops being possible to fail.
The difference is where the rule is enforced. Remember from Session 1 that every step produces a score for every token in the vocabulary. Constrained decoding inserts one operation between the scores and the sampling:
scores (logits) → mask illegal tokens to −∞ → probabilities (softmax) → sampleA tiny state machine tracks where you are in the schema. Just after {"total": it knows only a digit or a minus sign can legally come next, so every other token in the vocabulary has its score driven to negative infinity and its probability to exactly zero. Not discouraged — unreachable. The model is free to choose what the total is, and has no ability to choose a malformed shape.
A prompt requests
Format lives in text the model may or may not honour. Failures are rare, random, and therefore worst — they survive your testing and appear under load.
A schema enforces
Format lives in the decoder, so a malformed shape becomes unreachable — but truncation at max_output_tokens can still cut the JSON mid-object, so keep the try/except. Your format instructions also leave the prompt entirely, which frees tokens and attention for the actual task.
What it does not fix
Shape, not truth. A schema guarantees "total" is a number; it has no opinion on whether it's the right number. Grounding and evals remain your job.
One caution worth carrying to production: over-constraining can hurt quality. Force a terse schema on a genuinely hard question and you remove the model's room to reason before answering. The usual fix is to give the schema a "reasoning" field before the answer fields — order matters, because the model fills them in that order, and it can only condition on what it has already written.
Where vision quietly fails
"How many people in this photo?"
Crowd of 23. Model says 15. Or 30. Why?
Patches summarize — they don't enumerate. Counting is prediction, not tallying (same disease as 847×923 in Session 1). Fix: don't trust counts; use detection tools when counting matters.
You can't count either. One second on the clock — shout your number:
"What's to the LEFT of the pillar?"
Frequently confused. Why?
Patch positions are encoded only weakly — precise left/right/behind relations blur in the summary. Improving fast, still a weak spot.
Tiny or blurred text
A smudged total on a receipt. What does it do?
It may invent a plausible number rather than say "unreadable" — hallucination in pixels. Fix: Session 2's grounding line ("if unreadable, say so") works on images too. Test it in lab.
Faces and identity
"Who is this person?"
Models refuse identification by design (privacy). KYC face-match uses separate, purpose-built, regulated systems — not chat models.
Vision ideas that belong to Madurai
Inscription reader
Photo of temple Tamil inscriptions → transliteration + translation attempt.
Menu board decoder
Canteen board photo → structured menu + prices + "what's vegetarian?"
Lab-record digitizer
Handwritten observation tables → CSV. Your juniors will worship you.
Crop doctor
Leaf photo → likely disease + treatment, in Tamil, for a farmer's phone.
Mess-bill splitter
Receipt photo → items → who owes what.
Sign translator
Street sign photo → Tamil ⇄ English with context, for tourists.
Every one of these = the 4-line call you just saw + Session 2's prompting + an eval set. You learned everything they need before lunch — but the 4-line call is the prototype. What makes it a project: 20 labelled examples, a baseline to beat, and one documented failure mode.
Five ideas you own now
Patches
···
Images chop into squares → embeddings → the same attention loop. Visual words.
Diffusion
···
Learn to remove noise → start from pure noise → "repair" your way to a new image, steered by the prompt.
Speech
···
STT near-human, TTS convincing, cloning cheap — set a family password.
One call
···
contents=[image, question]. Prompting, format control and evals apply unchanged.
Pixels lie too
···
Counting, spatial precision, blurred text → plausible inventions. Ground it: "if unreadable, say so."
Lab 3: interrogate your photos
Stretch
Add the grounding line to your receipt prompt — does the invention stop? · Mini eval: 5 images with expected answers, score it (your S2 harness, now with eyes) · Audio: record a voice note, transcribe it.
No photo?
Sample images are linked in the notebook. But your own mess bill is more fun.
Day 1: you understand the machine.
Day 2: you arm it.
Today you built
First API calls → a prompt playbook → an eval harness that catches lies → vision apps over your own photos.
Tonight (5 minutes, not optional)
Put 2–3 real documents on your laptop or Drive: lecture notes, a textbook chapter PDF, anything you'd want to query. Tomorrow morning you build "chat with my notes" over them — the pattern behind most real AI products. No documents = boring capstone. Bring good ones.