RAG in Production: What the Tutorials Don't Tell You

The first version of our RAG pipeline took one afternoon. I embedded a handful of resumes, wired up a vector search, asked it questions, and got back answers that looked right. We demoed it internally that same week and people were genuinely impressed. Getting it in front of real recruiters took another four months, and almost none of that time went into anything a tutorial covers.

This was at Sagan World, where I led development of a recruitment platform with AI-reviewed interview screening. The system had to answer questions about candidates using their actual words: interview transcripts, resumes, written responses. The stakes were concrete. A hallucinated "five years of React experience" doesn't just look bad; it can cost a candidate a job or a client a bad hire.

Chunking is a product decision, not preprocessing

Every tutorial hands you fixed-size chunks, usually 512 tokens with some overlap, and moves on. That works on blog posts. On interview transcripts it was a quiet disaster. Chunks would start mid-answer, orphan a candidate's response from the question that prompted it, and split one anecdote across three chunks that each retrieved poorly on their own.

We rebuilt chunking around the structure of the documents themselves. Transcripts got chunked by speaker turn, with short turns merged and every answer kept attached to its question. Resumes went the opposite way: small, section-scoped chunks, because "skills" and "work history" answer very different questions. That one change moved retrieval quality more than any embedding model swap we tried, and we tried several.

The lesson that stuck with me: chunking encodes what your documents mean. It deserves the same design attention as your database schema.

Evaluate retrieval separately, or fly blind

For the first few weeks we tuned by vibes. Someone would report a bad answer, we'd poke at prompts, things would seem better, and a different failure would surface two days later. Classic whack-a-mole.

What fixed it was boring: a golden set. We collected about two hundred real recruiter questions and hand-labeled which transcript passages actually contained the answer. Then we measured recall@10 on every pipeline change. Our starting number was embarrassing, barely above 60%. Speaker-turn chunking plus hybrid search, BM25 alongside vectors in PostgreSQL with pgvector, pushed it close to 90%. Nothing about the LLM changed in that time.

Most of our "hallucination bugs" were retrieval bugs wearing a trench coat.

That was the biggest mindset shift. When the model invents things, your first suspect should be the retriever, because a model that never saw the right passage has no choice but to improvise. Build the eval set early. Ours took two days and paid for itself every week after.

Stale indexes and the plumbing nobody tweets about

Here's a failure mode no tutorial mentions: a recruiter edits a job description, the old embedding stays in the index, and the AI keeps screening candidates against requirements that no longer exist. Or a candidate re-uploads a corrected resume and both versions retrieve, contradicting each other.

Fixing this meant document versioning, embedding invalidation, a reindexing queue with idempotent workers, and per-tenant isolation so one client's data can never leak into another's retrieval. The "RAG" part of our FastAPI codebase was a few hundred lines. The freshness-and-correctness machinery around it ran into the thousands. That ratio is normal, and budgeting for it up front would have saved us a rough month.

Guardrails that actually held

We tried plenty of prompt-level "do not hallucinate" incantations. What actually worked was structural:

• Every claim in a review had to cite the chunk IDs it came from, and we verified those chunks were really retrieved in that request.
• If retrieval scores came back below a threshold, the system said "not mentioned in the interview" instead of answering.
• Anything numeric, from years of experience to salary expectations and notice periods, was extracted separately with strict validation rather than trusted from free-form generation.

The surprising part: recruiters trusted the system more after it started refusing. A confident wrong answer burns credibility that ten correct ones won't restore. "I couldn't find this" turned out to be a feature, not a failure.

If you're taking RAG to production, my honest advice after six years of shipping platforms and one bruising AI product: enjoy your afternoon demo, then budget a quarter for chunking, evals, freshness, and guardrails. That isn't the tax on the real work. It is the real work.