{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "colab": {
   "provenance": [],
   "name": "session_2_lab.ipynb"
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 2 — The Lie Detector\n",
    "**Session 2 · Prompt engineering + your first eval harness · TCE**\n",
    "\n",
    "You'll need: your **10-question file from Lab 1 Part E**.\n",
    "First: **File → Save a copy in Drive**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cell 1 — setup (same as Lab 1)\n",
    "%pip install -q -U google-genai\n",
    "from getpass import getpass\n",
    "from google import genai\n",
    "from google.genai import types\n",
    "import time\n",
    "\n",
    "client = genai.Client(api_key=getpass(\"Gemini API key: \"))\n",
    "MODEL = \"gemini-flash-latest\"  # the free tier's current Flash (July 2026 → Gemini 3.5 Flash). 503 'high demand'? swap to \"gemini-flash-lite-latest\".\n",
    "\n",
    "def ask(prompt, temperature=None):\n",
    "    config = types.GenerateContentConfig(temperature=temperature) if temperature is not None else None\n",
    "    for attempt in range(4):\n",
    "        try:\n",
    "            return client.models.generate_content(model=MODEL, contents=prompt, config=config).text\n",
    "        except Exception as e:\n",
    "            if \"429\" in str(e) and attempt < 3:\n",
    "                print(\"rate limited, waiting...\"); time.sleep(20*(attempt+1))\n",
    "            else: raise\n",
    "print(\"ready ✓\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part A — The prompt makeover (5 iterations)\n",
    "\n",
    "Below is a deliberately terrible prompt. Improve it **five times**, one upgrade per run:\n",
    "**v1** task (length+subject) → **v2** role+audience → **v3** context (real facts) → **v4** format+negative instructions → **v5** constraint (\"only stated facts\").\n",
    "\n",
    "Run, read, then edit `PROMPT` and run again. Document each step in the table below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cell 2 — edit PROMPT, run, repeat (keep old versions in comments!)\n",
    "PROMPT = \"write about tce\"   # v0 — terrible on purpose\n",
    "\n",
    "print(ask(PROMPT))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Document your makeover (edit this cell)\n",
    "\n",
    "| v | What I changed | Why the output got better |\n",
    "|---|---|---|\n",
    "| 1 | | |\n",
    "| 2 | | |\n",
    "| 3 | | |\n",
    "| 4 | | |\n",
    "| 5 | | |\n",
    "\n",
    "### ✓ Checkpoint 1 — five documented iterations.\n",
    "\n",
    "---\n",
    "## Part B — Your test set → your first eval\n",
    "\n",
    "Fill `my_tests` from your Lab 1 Part E file. Keep `expected` SHORT — the key fact only (a name, a number), not a full sentence. The scorer checks whether your expected string appears inside the model's answer (after normalization)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cell 3 — your 10 questions (2 examples shown — replace with YOURS)\n",
    "my_tests = [\n",
    "    {\"q\": \"Who composed the music for the film Roja?\", \"expected\": \"rahman\"},\n",
    "    {\"q\": \"In which year was TCE Madurai founded?\",     \"expected\": \"1957\"},\n",
    "    # ... add your 8+ more ...\n",
    "]\n",
    "print(len(my_tests), \"questions loaded\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cell 4 — the eval harness\n",
    "import re\n",
    "\n",
    "def norm(s):\n",
    "    return re.sub(r\"[^a-z0-9 ]\", \"\", s.lower())\n",
    "\n",
    "def run_eval(template, tests, verbose=True):\n",
    "    hits = 0\n",
    "    for t in tests:\n",
    "        ans = ask(template.format(q=t[\"q\"]), temperature=0.0)\n",
    "        ok = norm(t[\"expected\"]) in norm(ans)\n",
    "        hits += ok\n",
    "        if verbose:\n",
    "            print((\"✓\" if ok else \"✗\"), t[\"q\"])\n",
    "            if not ok:\n",
    "                print(\"   expected:\", t[\"expected\"], \"| got:\", ans[:120].replace(\"\\n\",\" \"))\n",
    "    score = hits / len(tests)\n",
    "    print(f\"\\nSCORE: {hits}/{len(tests)} = {score:.0%}\")\n",
    "    return score\n",
    "\n",
    "baseline = run_eval(\"Answer this question: {q}\", my_tests)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read every ✗ before moving on\n",
    "For each failure, decide: **model wrong** (hallucination — the interesting case), **scorer too strict** (fix your `expected` string), or **question ambiguous** (fix the question). This diagnosis IS the skill.\n",
    "\n",
    "### ✓ Checkpoint 2 — eval ran on your 10 questions; failures diagnosed.\n",
    "\n",
    "---\n",
    "## Part C — Prompt A vs Prompt B, settled with numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cell 5 — design a better template, then fight\n",
    "PROMPT_A = \"Answer this question: {q}\"\n",
    "\n",
    "PROMPT_B = (\n",
    "    \"You are a careful expert. Answer the question below.\\n\"\n",
    "    \"Rules: be direct, give the specific fact asked for, \"\n",
    "    \"and if you are not sure, say 'I am not sure' instead of guessing.\\n\\n\"\n",
    "    \"Question: {q}\\nAnswer:\"\n",
    ")   # ← edit me — beat A by more!\n",
    "\n",
    "print(\"=== A ===\"); score_a = run_eval(PROMPT_A, my_tests, verbose=False)\n",
    "print(\"=== B ===\"); score_b = run_eval(PROMPT_B, my_tests, verbose=False)\n",
    "print(f\"\\nA: {score_a:.0%}  vs  B: {score_b:.0%}  →  {'B wins' if score_b>score_a else 'A wins or tie — iterate B!'}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ✓ Checkpoint 3 — show me A vs B numbers + the single most interesting failure you found.\n",
    "\n",
    "---\n",
    "## Stretch goals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stretch 1 — variance: is your score stable?\n",
    "scores = [run_eval(PROMPT_B, my_tests, verbose=False) for _ in range(3)]\n",
    "print(\"three runs:\", [f\"{s:.0%}\" for s in scores], \"| average:\", f\"{sum(scores)/3:.0%}\")\n",
    "# We used temperature=0.0 in the harness — try changing it in run_eval and watch stability change."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stretch 2 — LLM-as-judge (a model grades a model)\n",
    "def judge_score(question, expected, answer):\n",
    "    verdict = ask(\n",
    "        f\"Question: {question}\\nExpected key fact: {expected}\\nStudent answer: {answer}\\n\"\n",
    "        \"Does the student answer contain the expected fact (paraphrase ok)? Reply only PASS or FAIL.\",\n",
    "        temperature=0.0)\n",
    "    return \"PASS\" in verdict.upper()\n",
    "\n",
    "t = my_tests[0]\n",
    "ans = ask(PROMPT_B.format(q=t[\"q\"]))\n",
    "print(\"judge says:\", judge_score(t[\"q\"], t[\"expected\"], ans))\n",
    "# Now: where might the judge itself be wrong? (verbosity bias, self-agreement...)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrap\n",
    "You now own the loop: **prompt → eval → read failures → fix → re-run.** Keep this notebook — the same harness grades your capstone in Session 6.\n",
    "\n",
    "**Short break. Session 3: AI gets eyes and ears — have a photo or two on your phone.**"
   ]
  }
 ]
}