Giving AI
your knowledge
The model has read the internet — but not your notes, your syllabus, your company's documents. Today: search that understands meaning, then RAG — the architecture inside nearly every "chat with your docs" product ever shipped. By lunch you'll have built "chat with my notes."
Did it survive the night?
Watch it not know — loudly
It cannot know — your syllabus isn't in its training data. Watch what it does instead of admitting that: a plausible OS syllabus, invented on the spot. Some models even apologise first — then produce a typical syllabus anyway. The hedge is not knowledge. Session 2's disease, aimed directly at your college life — today we cure it.
"Just paste all my notes!" — let's price that
Three taxes on pasting
1. The window — remember, everything must fit (Session 1). A textbook doesn't.
2. The meter — you pay per token, per question. Every "what's a deadlock?" re-sends all 400 pages.
3. The middle — even long-window models attend unevenly; whatever's buried mid-window is likeliest to be missed. More haystack, worse needle-finding.
Don't send everything.
Send the right 3 paragraphs.
Nobody reads the whole book in an open-book exam — toppers walk in with sticky flags already on the right pages. That's a good index. We need a search engine that finds paragraphs by meaning, staples them to the question, and lets the model answer from them.
Retrieval → Augmentation → Generation. RAG. Let's build the R first.
Keyword search misses meaning
Zero shared words — "marks/clear/subject" vs "50% aggregate/internals". Keyword search: nothing. But the meanings are neighbours… and you already know a machine that maps meaning to coordinates.
Embeddings, now for whole paragraphs
Same trick, bigger units
Session 1: words → coordinates (idli next to dosa). Today: entire chunks of text → coordinates. "Minimum 50% aggregate to pass" and "marks required to clear" land near each other in meaning-space — zero shared words needed.
Search = nearest neighbours
Embed all your chunks once (cheap). Embed the question. Find the chunks whose vectors sit closest — cosine similarity, one line of numpy. That's semantic search. That's the R in RAG.
Three things are quietly doing work in chunk_vectors @ question_vector, and each is a decision you could get wrong.
1 · A paragraph has many tokens but one vector. Session 1 gave every token coordinates. An embedding model runs the same transformer and then pools — usually averaging all the token vectors, sometimes taking one designated summary token — into a single vector for the whole passage. That's the compression that makes search possible, and also its main limitation: a 400-word chunk covering three topics gets one blurred point, sitting near none of them. It is a real argument for smaller chunks.
2 · The matrix multiply is cosine only because the vectors are normalized. Cosine similarity is the dot product divided by both lengths. Scale every vector to length 1 up front and the divisions become 1 forever, so similarity collapses into plain multiply-and-add — which is why a numpy one-liner searches thousands of chunks instantly. Forget the normalization and you silently rank by length as much as by meaning.
cos(a,b) = (a·b)/(|a||b|) → a·b when |a| = |b| = 13 · A question is not a document, and good embedding APIs know it. "How many marks to pass?" and the paragraph that answers it don't look alike — one is short and interrogative, the other long and declarative. Modern embedding models therefore accept a task type, embedding queries and documents slightly differently so they land near each other anyway. Getting this backwards is a quiet, hard-to-spot accuracy leak.
Dimensions are a dial
768 numbers per chunk is a default, not a law. Many current models are trained so you can truncate the vector — keep the first 256 — and lose surprisingly little. Storage and search time drop proportionally.
Never mix models
Vectors from two different embedding models are not comparable — different spaces entirely. Change the model and you must re-embed every chunk. Store the model name beside the index or you will eventually debug this for a day.
Embed once, query forever
Indexing cost is one-off and cheap; query cost is per-question and tiny. This asymmetry is the reason RAG beats re-pasting your documents on every single call.
The search playground
Chunking: how you cut the book
Too small → the retrieved fragment lacks context. Too big → the right sentence drowns, similarity dilutes. Paragraph-with-overlap is the boring default that wins. Chunking bugs cause more RAG failures than model choice does.
Build the boring default first — you will in the lab. But when it plateaus, these are the moves in roughly the order they pay off, and none of them require a framework.
Respect structure
Split on the document's own hierarchy — headings, then paragraphs, then sentences — recursing only when a piece is still too big. A chunk that stops mid-table or mid-clause was destroyed before it was ever embedded. This alone beats most tuning.
Small to search, big to read
Index small precise chunks, but when one wins, feed the model its parent section. You get the retrieval accuracy of small chunks and the context of large ones. Cheapest real upgrade in RAG.
Carry metadata
Store source, section, page and date with each chunk. It powers citations, lets you filter before you search ("only 2026 regulations"), and makes stale content findable. Free at index time, impossible to add later.
Prepend context
Give each chunk a one-line header of where it came from — document title and section — before embedding. "…and the end-semester exam" becomes searchable when it carries "Attendance Rules ›" on its front.
Filter before you rank, not after. If the user asks about this year's syllabus, a metadata filter on year narrows the candidates first, and semantic search then ranks within a set that is already correct. Doing it the other way round means competing against every similar-looking paragraph from every previous year — a large share of "the retrieval is bad" complaints are really a missing filter.
A test that costs nothing and finds most chunking bugs: print ten random chunks and read them cold. If you can't tell what a chunk is about without seeing the original document, neither can the embedding model — and no amount of prompt engineering downstream will recover it.
Vector databases, in plain words
What it is
A library shelved by meaning — hands back the k nearest vectors to any query, fast, even across millions.
Names you'll meet
FAISS (library), Chroma (dev-friendly), pgvector (Postgres extension), Pinecone/Weaviate (managed). Same idea, different packaging — like filter coffee from a tumbler, a flask, or a ₹300 café cup.
What YOU need today
A numpy array. Truly. Below ~100k chunks, one argsort is instant. Don't add a database until you have a database-sized problem.
Resume line, unlocked: "built semantic search over a vector store" — that sentence is the numpy array you'll write in the next 20 minutes.
RAG, end to end
Your pipeline retrieves the top 3 and sends them. Every serious RAG system in 2026 does something slightly different, and it's the single highest-leverage upgrade you can make after chunking: retrieve wide, then re-rank narrow.
retrieve top 20 (fast, approximate) → rerank to the best 4 (slow, accurate) → augment → generateWhy two stages instead of one good one? Because your embedding search is a bi-encoder: it embedded the question and every chunk separately, so the two never met — that's exactly what makes it fast enough to search a million chunks, since all the document work happened at index time. A cross-encoder reranker instead reads the question and one chunk together and scores their actual relevance. Far more accurate, far too slow to run over a whole corpus — perfect for re-ordering 20 candidates.
The measurable win is usually large, because your first stage is better at recall than at ranking: the right chunk is frequently sitting at position 9 when you only took 3. Reranking doesn't find anything new — it stops you from throwing away what you already found.
Hybrid search
Run keyword (BM25) and semantic search side by side and merge the two ranked lists. Keyword catches exact identifiers embeddings blur — course codes, error numbers, surnames, "Section 4(b)". Semantic catches the paraphrases keyword misses. Neither is redundant.
Query rewriting
Ask the model to expand the question before you search it — resolve the pronouns from the chat history, add synonyms, split a two-part question into two searches. Fixes the vocabulary gap on slide 13 at its source.
A reranker in one call
No new service needed: pass the 20 candidates to a cheap fast model and ask it to return the 4 most relevant ids as JSON, with a schema. Slower than a dedicated reranker, dramatically better than nothing, and about ten lines. This is a lab stretch.
The trap to avoid: adding all of this on day one. Build the simple pipeline, measure it, and add a stage only when your eval shows the right chunk was retrieved but ranked too low. Every stage is latency, cost, and one more thing that can break — and Session 5 will show you what happens to reliability when you chain steps without measuring.
The grounded prompt template
Three load-bearing lines
ONLY the context — blocks its internet-memories from overriding your documents.
Cite the chunk — users can verify; hallucinations become visible.
"I don't know" escape hatch — gives it a legal way out. Without one, it fills silence with fiction (you watched this at 9 a.m.).
Where RAG breaks in the wild
Right answer, wrong words
The user asks "attendance shortage rules" — notes say "condonation policy". Retrieval whiffs. Why?
Vocabulary gap even embeddings can miss. Fix: rephrase the query with the LLM first, retrieve more chunks (k=5), or index chunk summaries too.
The answer got cut in half
Rule starts in chunk 7, exception lives in chunk 8. Model sees only chunk 7. Result?
Confidently incomplete answer. Fix: overlapping chunks, or retrieve neighbours of every hit.
Stale index
Syllabus PDF updated Monday; embeddings made last month. What happens?
Confident answers from the OLD syllabus — with citations! RAG trusts its shelf. Fix: re-index on change; store doc dates; show them in answers.
Model ignores your context
Context says internal exam is 25 marks; the internet-average says 20. Model answers 20. Why?
Training memories leak past weak grounding. Fix: the ONLY line, lower temperature, and put context BEFORE the question. Test it — that's an eval (Session 2 never left).
Notice that the four failures above split cleanly into two groups. The first two are retrieval failures — the right text never arrived. The last two are generation failures — the right text arrived and the model mishandled it. One overall "is the answer good?" score cannot tell them apart, which is why teams stare at a mediocre number for a week and tune the wrong end of the pipeline.
So score the two halves separately. For each question in your test set, record which chunk should win.
Recall@k — did it arrive?
In what fraction of questions is the correct chunk somewhere in the top k? This is your ceiling: if recall@5 is 60%, no prompt on earth gets you past 60% correct answers. Fix it with chunking, hybrid search, or query rewriting.
MRR — did it arrive near the top?
Average of 1/(rank of the first correct chunk). Rank 1 scores 1.0, rank 5 scores 0.2. High recall with low MRR is the exact signature that says add a reranker — you're finding it and then burying it.
Faithfulness — did it stick to it?
Given that the right chunk was supplied, did the answer come from it? Check the citation actually supports the sentence. Low faithfulness is a grounding and prompt problem, never a retrieval one.
Read them in that order and the diagnosis is nearly automatic. Low recall → your ingest is broken; nothing downstream will save you. Good recall, poor MRR → your ranking is broken; rerank. Good retrieval, poor faithfulness → your prompt is broken; strengthen the ONLY line, drop the temperature, put context before the question.
There is also a failure mode no score catches: the answer is simply not in your documents. The correct behaviour is the escape hatch — "I don't know based on the provided documents" — so put a few unanswerable questions in your test set on purpose and check the system refuses. A RAG app that never says "I don't know" isn't confident, it's uncalibrated.
RAG vs paste-it-all vs fine-tuning
| Approach | When it wins | When it loses |
|---|---|---|
| Paste into context | Few pages, one-off questions, quick scripts | Big corpora · repeated queries (you re-pay every time) |
| RAG | Large or changing knowledge · need citations · need freshness | Tiny static docs (overkill) · answers needing whole-corpus reasoning |
| Fine-tuning | Style, format, domain behaviour at scale | Teaching facts — expensive, freezes instantly, no citations. Common ₹-crore mistake. |
Interview one-liner: "Fine-tuning teaches behaviour; RAG provides knowledge." Full decision ladder after lunch.
You've been using RAG all along
Support chatbots
"Chat with our help docs." Retrieval over a manual + the grounded template.
NotebookLM-style tools
"Chat with your sources" — RAG with a polished UI.
AI search engines
Search results → context → cited answer. RAG at web scale.
Legal/medical assistants
Retrieval over case law / literature — citations are mandatory there.
"Chat with our wiki"
Every enterprise AI rollout starts here. Somewhere in Bengaluru, someone's entire salary is today's lab.
Yours, in 50 minutes
Chat with your own notes — same architecture, your data.
Your final-year project is
one grounded prompt away from being a startup.
Most funded "AI products" are exactly what you build today: retrieval plus a grounded prompt around someone else's model. That is not an insult to them — it is an invitation to you. The moat is the data and the evals, and you have Madurai data nobody in San Francisco has.
Disagree? Good. Bring it to the break — strongest counter-argument gets named on the closing slide.
Lab architecture: two pipelines
~60 lines of Python total. Every stage is something you've already touched this weekend.
Six ideas you own now
Chunk embedding
···
Whole paragraphs → coordinates. Meaning-neighbours, zero shared words needed.
Semantic search
···
Embed chunks once + question → cosine similarity → top-k. One numpy line.
Chunking
···
Paragraph + overlap. Too small = no context; too big = diluted. Causes more failures than model choice.
Augmentation
···
Staple top chunks into the grounded template: ONLY the context · cite · "I don't know" escape hatch.
RAG vs fine-tune
···
Fine-tuning teaches behaviour; RAG provides knowledge. Facts → RAG.
Fresh index
···
RAG trusts its shelf — stale embeddings give confident, cited, outdated answers. Re-index on change.
Lab 4: chat with YOUR notes
Stretch
Mini-eval: 5 questions with expected answers → score your RAG (harness from S2!) · k=1 vs k=3 vs k=5 · second document · show similarity scores in answers.
Capstone alert
This app is your capstone foundation. This afternoon adds tools (S5) and hardening + demo (S6). Build it on documents you actually care about.
Your AI now knows
what you know.
Next: it stops just answering and starts doing — calculators, search, your files. Tool use, agents, and the honest truth about both.