AI beyond text:
eyes, ears and a paintbrush
Vision that reads your mess bill, pictures conjured out of pure static, voices you can't tell from human — and all of it still the one loop you met this morning.
Still true after the break?
Same loop.
New kinds of tokens.
Session 1's machine never cared what tokens meant. So: chop an image into little squares, turn each into an embedding — the attention loop reads a photo the way it reads a sentence. Chop audio into ~20-ms slices — it reads sound.
"Multimodal" isn't a new brain. It's new eyes wired into the same brain.
How a model reads a picture
Patches are visual words
The image splits into a grid of small squares. Each patch → an embedding → the same attention loop from Session 1. "What's in this photo?" is literally next-token prediction with picture-tokens in the context.
This is why vision arrived so fast once transformers worked — the hard part was already built.
It doesn't just see. It reads.
Documents
Receipts, invoices, mark sheets, forms → clean JSON. The most-shipped, least-glamorous vision use-case on the planet.
Handwriting
Your lab-record pages, doctor scrawl (mostly), whiteboard photos → text.
Charts & screens
Reads graphs, tables, app screenshots — "what's wrong with this UI?" works.
KYC pipelines
ID extraction + face match + tamper checks — how fintechs onboard you in minutes. This is my day job.
Accessibility
Live alt-text for blind users — describing the visual web. Quietly life-changing.
Field work
Leaf-disease detection from a farmer's phone photo. No lab needed.
Will it read it? Place your bets
Reading pictures is prediction.
Making them is un-destruction.
Most image generators aren't next-token machines. They use a different, equally simple trick: learn to remove noise from photos… then hand them pure noise and let them "remove" their way to a picture that was never there.
It's called diffusion. Watch it happen →
(One 2025 twist: OpenAI's 4o images are generated token-by-token, autoregressively — the loop ate images too. Diffusion is still the concept to know.)
Diffusion: a picture emerges from static
Training: destroy, then learn to repair
Take billions of real images, add noise step by step until pure static, and train a network to reverse each step.
Generation: start from fresh random static and apply the repair over and over. The "repair" invents a plausible image — steered at every step by your text prompt ("a gopuram at sunset"), which is embedded with the same trick from Session 1.
Our canvas fakes the pixels, honestly — but the choreography is true: coarse shapes resolve first, details last. Silhouette by step ~15, sun by ~30, window by ~45.
Image generation: magic with fine print
Details lie
The six-finger memes are fixed. The subtle stuff isn't: dense small text, exact logos, jewellery, background faces. Zoom in before you trust.
Style questions
Trained on human art. "In the style of a living artist" raises real ethical and legal debates — unresolved, ongoing.
Deepfake era
If images can be conjured, images stop being evidence. Provenance tools (watermarking, content credentials) are the counter-move.
The camera spent 200 years earning our trust. Diffusion ended it in two. Same lesson as Session 2: the output is plausible, not verified — now in pixels.
Speech: solved enough to be dangerous
Speech → text
Near-human transcription, dozens of languages, decent Tamil-English code-switching. Lectures → notes is a solved problem.
Text → speech
Voices now genuinely hard to tell from human — tone, pauses, emotion.
Voice cloning
A few seconds of audio can clone a voice. "Your grandson" calling paati for an OTP is a live scam pattern in India right now — agree on a family password. Seriously. Tell your parents this weekend.
Video: the same diffusion family, plus time. Mid-2026 reality: native 4K clips with the soundtrack generated alongside the pixels (Sora 2, Veo 3.1, Kling 3.0). Physics still slips in complex scenes, and it costs real money — roughly $0.10–0.75 per second. Improving every quarter — what you're watching today is the worst it will ever be.
Your mother’s voice is no longer
proof of your mother.
Three seconds of audio clones a voice well enough to fool family — voice-clone scam calls already run at scale in India. The fix is not better ears; it is verification habits: a family code word beats a trained ear.
Disagree? Good. Bring it to the break — strongest counter-argument gets named on the closing slide.
All of it is one API call
The contents list mixes images and text freely. Audio files work the same way. Everything from Sessions 1–2 (prompting, format control, evals!) applies unchanged.
The production way: you don't beg for JSON in the prompt — you pass response_schema and the API is forced to return valid JSON matching your shape. In the lab you'll do it both ways — then delete half your prompt.
Where vision quietly fails
"How many people in this photo?"
Crowd of 23. Model says 15. Or 30. Why?
Patches summarize — they don't enumerate. Counting is prediction, not tallying (same disease as 847×923 in Session 1). Fix: don't trust counts; use detection tools when counting matters.
You can't count either. One second on the clock — shout your number:
"What's to the LEFT of the pillar?"
Frequently confused. Why?
Patch positions are encoded only weakly — precise left/right/behind relations blur in the summary. Improving fast, still a weak spot.
Tiny or blurred text
A smudged total on a receipt. What does it do?
It may invent a plausible number rather than say "unreadable" — hallucination in pixels. Fix: Session 2's grounding line ("if unreadable, say so") works on images too. Test it in lab.
Faces and identity
"Who is this person?"
Models refuse identification by design (privacy). KYC face-match uses separate, purpose-built, regulated systems — not chat models.
Vision ideas that belong to Madurai
Inscription reader
Photo of temple Tamil inscriptions → transliteration + translation attempt.
Menu board decoder
Canteen board photo → structured menu + prices + "what's vegetarian?"
Lab-record digitizer
Handwritten observation tables → CSV. Your juniors will worship you.
Crop doctor
Leaf photo → likely disease + treatment, in Tamil, for a farmer's phone.
Mess-bill splitter
Receipt photo → items → who owes what.
Sign translator
Street sign photo → Tamil ⇄ English with context, for tourists.
Every one of these = the 4-line call you just saw + Session 2's prompting + an eval set. You learned everything they need before lunch.
Five ideas you own now
Patches
···
Images chop into squares → embeddings → the same attention loop. Visual words.
Diffusion
···
Learn to remove noise → start from pure noise → "repair" your way to a new image, steered by the prompt.
Speech
···
STT near-human, TTS convincing, cloning cheap — set a family password.
One call
···
contents=[image, question]. Prompting, format control and evals apply unchanged.
Pixels lie too
···
Counting, spatial precision, blurred text → plausible inventions. Ground it: "if unreadable, say so."
Lab 3: interrogate your photos
Stretch
Add the grounding line to your receipt prompt — does the invention stop? · Mini eval: 5 images with expected answers, score it (your S2 harness, now with eyes) · Audio: record a voice note, transcribe it.
No photo?
Sample images are linked in the notebook. But your own mess bill is more fun.
Day 1: you understand the machine.
Day 2: you arm it.
Today you built
First API calls → a prompt playbook → an eval harness that catches lies → vision apps over your own photos.
Tonight (5 minutes, not optional)
Put 2–3 real documents on your laptop or Drive: lecture notes, a textbook chapter PDF, anything you'd want to query. Tomorrow morning you build "chat with my notes" over them — the pattern behind most real AI products. No documents = boring capstone. Bring good ones.