Talking to AI,
and catching its lies
Prompts that actually work — and the engineer's answer to "how do you know it's right?": you measure. Today you build the measuring instrument.
Still true after the break?
Two artifacts in two hours
A prompt playbook
Role, task, context, format, examples, constraints — you'll take one terrible prompt and rebuild it five times, watching quality climb each step.
An eval harness
The 10 expert questions you wrote in Lab 1 become a test set. Your code will score the model against it — and score your prompts against each other. This one artifact is the difference between using AI and engineering it.
Every AI team on Earth ships on the same loop: prompt → measure → fix → repeat. Today you run your first lap.
The prompt makeover: five upgrades
Anatomy of a prompt: six switches
Task is mandatory. Everything else is a dial — turn on what the job needs. Over-stuffing costs tokens and dilutes attention (you know why now).
Few-shot: show, don't tell
Examples pin down format, labels and edge cases better than any description. The model is an imitation engine — so hand it something worth imitating. 2–5 good examples out-argue a page of instructions.
Step-by-step beats straight-to-answer
Steps force the arithmetic into visible tokens you can check — errors surface instead of hiding. Modern models often reason internally — but when you must audit the logic (money, marks, medicine), asking for the working still wins.
Demand a format, or parse chaos forever
Apps don't read prose. The word ONLY is doing real work — without it you get JSON wrapped in “Sure! Here's your data:” and your parser dies at 2 a.m.
The four classic prompt crimes
The kitchen sink
"Summarize this, also translate it, also make a quiz, also suggest improvements, also…"
One prompt, one job. Chain simple calls instead — each does one thing well, and you can test each. (This is Session 5's workflow idea, arriving early.)
Vague adjectives
"Make it good. Make it professional. Make it pop."
Define it or show it. "Professional" = whose? Give a rubric ("no slang, ≤3 sentences, active voice") or 2 examples of what good looks like.
Assuming it remembers
"Like I told you yesterday…" / "use my usual style"
It remembers nothing (Session 1!). Everything needed must be in THIS context window. Re-send the facts, every time.
No format spec
Then spending an hour parsing "Certainly! Here are some thoughts…"
Specify structure up front: JSON schema, bullet limit, word cap, "reply ONLY with…". Your parser will thank you.
Your polished prompt
still lies beautifully.
A US lawyer submitted a brief citing six court cases — ChatGPT invented all six. He was sanctioned in federal court. Air Canada's chatbot promised a refund policy that didn't exist — a tribunal made the airline honour it.
Mata v. Avianca (S.D.N.Y., 2023) — six fabricated cases, $5,000 sanction · Moffatt v. Air Canada (2024 BCCRT 149) — chatbot invented a refund policy, airline held liable
These weren't bad prompts. They were unmeasured ones.
One of these is a confident lie
The scary part isn't that it lies — it's that the lie is indistinguishable in tone from the truths. No amount of squinting at style detects it. Only checking does.
Hallucination is not a bug.
It is the model doing exactly what it was trained to do.
It was trained to produce plausible text, not true text. Plausibility was the objective; truth was never in the loss function. Accept that, and "how do I stop it lying?" becomes an engineering question — with engineering answers.
Disagree? Good. Bring it to the break — strongest counter-argument gets named on the closing slide.
"How do you know it's right?" You measure.
Test set
Questions with known correct answers — written by someone who actually knows. (You, in Lab 1 Part E.)
Scorer
Code that compares the model's answer to the expected one and returns ✓ or ✗. No human squinting.
Score
One number. Now "is prompt B better?" has an answer instead of an argument.
"In God we trust; all others must bring data." — the engineer's oldest prayer, today pointed at AI. Or in this course's dialect: a prompt without an eval is a superstition.
Watch an eval run
Read the failures, not the score
7/10 isn't the insight. Which three failed, and why — that's the insight. Wrong facts? Format drift? Too-strict matching? Each failure type has a different fix.
The score is a dice roll
Run it twice — the score moves. Session 1's dice, back again. The discipline: temperature 0, three runs, report the average and the spread — even T=0 isn't perfectly deterministic in practice; batching and floating point wobble it.
Same answer, three verdicts
The metric is a design decision. Too strict → punishes correct answers. Too loose → rewards garbage. Your scorer is part of your system — and sometimes the scorer is the liar.
Prompt A vs Prompt B: the arena
say I-don't-know otherwise" + format
"It worked when I tried it"
is not evidence.
You tried 3 questions. Users will bring 3,000 — misspelled, in Tanglish, about edge cases you never imagined. The demo shows the best case; the eval shows the expected case.
Every AI product that embarrassed its company in the news had a great demo first.
Eval-driven development: the loop
Change one thing at a time or you'll never know what worked. Failures aren't embarrassments. They're the syllabus.
Six ideas you own now
Prompt anatomy
···
Role · Task · Context · Format · Examples · Constraints. Task mandatory, rest are dials.
Few-shot
···
2–5 examples pin format and edge cases. The model imitates — feed it patterns.
Step-by-step
···
Visible reasoning you can audit. Errors surface in the working.
Hallucination
···
Confident, fluent, wrong — indistinguishable by tone. Only checking detects it.
Test set
···
Questions + expected answers + scorer = one honest number.
The loop
···
Prompt → eval → read failures → fix one thing → re-run. T=0, ×3, average.
Lab 2: the lie detector
Stretch
LLM-as-judge scorer · run ×3 and report the spread · does temperature change your score?
You can now prove
whether AI is right.
Next session: AI grows eyes and ears — images, documents, voice. Have a photo or two on your phone — you'll interrogate them.