{
 "nbformat": 4,
 "nbformat_minor": 5,
 "metadata": {
  "colab": {
   "provenance": [],
   "name": "session_3_lab.ipynb"
  },
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Lab 3 — Interrogate Your Photos\n",
    "**Session 3 · Multimodal · TCE** · First: **File → Save a copy in Drive**.\n",
    "\n",
    "You need 2–3 photos: anything on your phone — a receipt, your handwritten notes, the canteen menu board. Transfer to laptop (email yourself / Drive / USB cable)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cell 1 — setup\n",
    "%pip install -q -U google-genai pillow\n",
    "from getpass import getpass\n",
    "from google import genai\n",
    "from google.genai import types\n",
    "from PIL import Image\n",
    "import time\n",
    "\n",
    "client = genai.Client(api_key=getpass(\"Gemini API key: \"))\n",
    "MODEL = \"gemini-flash-latest\"  # the free tier's current Flash (July 2026 → Gemini 3.5 Flash). 503 'high demand'? swap to \"gemini-flash-lite-latest\".\n",
    "\n",
    "def _shrink(x, cap=1024):\n",
    "    \"\"\"Downscale big images before sending — saves free-tier quota and time.\n",
    "    A 4000px phone photo and a 1024px one give the model the same answer.\"\"\"\n",
    "    if isinstance(x, Image.Image) and max(x.size) > cap:\n",
    "        r = cap / max(x.size)\n",
    "        return x.resize((int(x.width*r), int(x.height*r)))\n",
    "    return x\n",
    "\n",
    "def ask(contents, temperature=None):\n",
    "    \"\"\"contents can be a string, or a list mixing PIL Images and strings.\"\"\"\n",
    "    if isinstance(contents, list):\n",
    "        contents = [_shrink(c) for c in contents]\n",
    "    config = types.GenerateContentConfig(temperature=temperature) if temperature is not None else None\n",
    "    for attempt in range(4):\n",
    "        try:\n",
    "            return client.models.generate_content(model=MODEL, contents=contents, config=config).text\n",
    "        except Exception as e:\n",
    "            if \"429\" in str(e) and attempt < 3:\n",
    "                print(\"rate limited...\"); time.sleep(20*(attempt+1))\n",
    "            else: raise\n",
    "print(\"ready ✓\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part A — Photo Q&A (escalating difficulty)\n",
    "\n",
    "Upload a photo: **folder icon (left sidebar) → upload**. Then run the ladder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cell 2 — the interrogation ladder\n",
    "img = Image.open(\"your_photo.jpg\")   # ← your filename\n",
    "display(img.resize((min(400, img.width), int(img.height * min(400, img.width) / img.width))))\n",
    "\n",
    "questions = [\n",
    "    \"Describe this image in 2 sentences.\",\n",
    "    \"Read ALL text visible in this image, exactly as written.\",\n",
    "    \"How many distinct objects/people are in this image? Count carefully.\",\n",
    "    \"What can you infer about where and when this was taken?\",\n",
    "    \"What is the most surprising detail in this image?\",\n",
    "]\n",
    "for q in questions:\n",
    "    print(\"=\" * 60, \"\\nQ:\", q)\n",
    "    print(ask([img, q]), \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Grade it (edit this cell)\n",
    "Which answers were right? Where did it wobble — counting? small text? inference?\n",
    "\n",
    "### ✓ Checkpoint 1 — ladder run + your 2-line grading.\n",
    "\n",
    "---\n",
    "## Part B — Receipt / document → JSON\n",
    "\n",
    "Session 2's format control, now with eyes. Strict schema, `ONLY`, grounding line."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cell 3 — structured extraction\n",
    "doc = Image.open(\"receipt.jpg\")   # ← receipt / bill / marksheet / form\n",
    "\n",
    "schema_prompt = \"\"\"Extract data from this image.\n",
    "Reply ONLY with JSON in exactly this schema:\n",
    "{\"vendor\": str, \"date\": str, \"items\": [{\"name\": str, \"price\": float}], \"total\": float}\n",
    "If any field is unreadable, use null — do NOT guess.\"\"\"\n",
    "\n",
    "print(ask([doc, schema_prompt], temperature=0.0))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cell 4 — prove it parses (the real test)\n",
    "import json as pyjson\n",
    "raw = ask([doc, schema_prompt], temperature=0.0)\n",
    "# strip accidental code fences if present\n",
    "raw = raw.strip().removeprefix(\"```json\").removeprefix(\"```\").removesuffix(\"```\").strip()\n",
    "data = pyjson.loads(raw)\n",
    "print(\"PARSED ✓  total =\", data[\"total\"])\n",
    "# If this cell crashes, your prompt isn't strict enough. Tighten and re-run — that's the lesson."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ✓ Checkpoint 2 — `json.loads` succeeds on your document."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Part B2 — the production way: guaranteed JSON\n",
    "\n",
    "Begging for JSON in the prompt works — you just proved it. Production code doesn't beg: it passes a **schema** with the request. The API then **cannot** return anything else — no code fences, no \"Certainly!\", nothing to strip — and you can delete the format instructions from your prompt entirely. The prompt says *what* to extract; the schema locks the *shape*."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cell 4b — response_schema: the API cannot answer in anything but your JSON\n",
    "schema = {\n",
    "    \"type\": \"OBJECT\",\n",
    "    \"properties\": {\n",
    "        \"vendor\": {\"type\": \"STRING\"},\n",
    "        \"date\":   {\"type\": \"STRING\"},\n",
    "        \"total\":  {\"type\": \"NUMBER\"},\n",
    "        \"items\":  {\"type\": \"ARRAY\", \"items\": {\n",
    "            \"type\": \"OBJECT\",\n",
    "            \"properties\": {\"name\": {\"type\": \"STRING\"}, \"price\": {\"type\": \"NUMBER\"}},\n",
    "            \"required\": [\"name\", \"price\"],\n",
    "        }},\n",
    "    },\n",
    "    \"required\": [\"vendor\", \"total\"],\n",
    "}\n",
    "\n",
    "resp = client.models.generate_content(\n",
    "    model=MODEL,\n",
    "    contents=[_shrink(doc), \"Extract the receipt.\"],   # ← no format instructions at all\n",
    "    config=types.GenerateContentConfig(\n",
    "        response_mime_type=\"application/json\",\n",
    "        response_schema=schema,\n",
    "    ),\n",
    ")\n",
    "data = pyjson.loads(resp.text)   # no fence-stripping, no crash — ever\n",
    "print(\"PARSED ✓  vendor =\", data[\"vendor\"], \"· total =\", data[\"total\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Part C — Handwriting test\n",
    "\n",
    "Photograph a page of YOUR handwritten notes → transcribe → grade yourself: roughly what % correct? Tamil/Tanglish notes = bonus experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cell 5 — handwriting\n",
    "notes = Image.open(\"my_notes.jpg\")\n",
    "print(ask([notes, \"Transcribe this handwritten page exactly. Mark unclear words as [?].\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part D — Break it\n",
    "\n",
    "Find one image where the model **confidently invents a detail**: a blurred price it \"reads\" anyway, objects it miscounts, text it paraphrases instead of quoting.\n",
    "\n",
    "### ✓ Checkpoint 3 — show me the invention.\n",
    "\n",
    "---\n",
    "## Stretch goals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stretch 1 — does grounding stop the invention?\n",
    "# Re-run your Part D image WITH the grounding line vs WITHOUT:\n",
    "loose  = \"What is the total on this receipt?\"\n",
    "strict = \"What is the total on this receipt? If it is not clearly readable, reply exactly: UNREADABLE.\"\n",
    "img_d = Image.open(\"your_partD_image.jpg\")\n",
    "print(\"loose :\", ask([img_d, loose]))\n",
    "print(\"strict:\", ask([img_d, strict]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stretch 2 — vision eval (your S2 harness grows eyes)\n",
    "vision_tests = [\n",
    "    {\"img\": \"receipt.jpg\", \"q\": \"What is the total?\", \"expected\": \"342\"},\n",
    "    # add 4 more: photo, question, expected key fact\n",
    "]\n",
    "import re\n",
    "def norm(s): return re.sub(r\"[^a-z0-9 ]\", \"\", s.lower())\n",
    "hits = 0\n",
    "for t in vision_tests:\n",
    "    ans = ask([Image.open(t[\"img\"]), t[\"q\"]], temperature=0.0)\n",
    "    ok = norm(t[\"expected\"]) in norm(ans); hits += ok\n",
    "    print(\"✓\" if ok else \"✗\", t[\"q\"], \"→\", ans[:60])\n",
    "print(f\"vision score: {hits}/{len(vision_tests)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Stretch 3 — audio: record a short voice note on your phone, upload it\n",
    "audio = client.files.upload(file=\"voicenote.m4a\")\n",
    "print(ask([audio, \"Transcribe this audio, then summarize it in one line.\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Day 1 complete\n",
    "\n",
    "**Tonight (5 min, mandatory):** put 2–3 real documents (lecture notes, textbook chapter PDF) on your laptop/Drive. Tomorrow: **chat with your notes** — the pattern behind most real AI products.\n",
    "\n",
    "Sleep well. Day 2 is the good stuff."
   ]
  }
 ]
}