A practical RAG production checklist

1. Corpus and freshness

• Who owns source documents, and how do updates reach the index?
• Are access controls preserved through chunking and retrieval?
• Do you know stale-data failure modes for your domain?

2. Retrieval quality

• Golden questions with expected passages — not only vibe checks.
• Hybrid search when keywords and semantics both matter.
• Citation UX so users (and auditors) can verify answers.

3. Generation and safety

• Clear “I don’t know” behavior when retrieval is weak.
• Prompt and tool boundaries for regulated domains (health, finance).
• Latency and cost budgets under realistic concurrency.

4. Evaluation and monitoring

• Offline eval set plus online feedback loop.
• Logging of queries, retrieved docs, and refusals (with privacy review).
• Owner on-call when quality drifts after a content or model change.

5. Ownership

If nobody owns corpus freshness, retrieval quality, and incident response, you do not have a production RAG system — you have a demo with a longer timeout.

← All writing · Work with me on RAG systems