Problem
Healthcare operations knowledge lives in PDFs, scanned forms, and shifting policy docs. Teams waste time hunting answers — and naive LLM chat invents details that cannot survive audit. The need was a path from messy documents → grounded answers.
My role
- Designed the document → chunk → index → retrieve → generate loop with grounding checks.
- Chose parsing and chunking strategies for mixed digital/scanned healthcare artifacts.
- Defined evaluation for citation fidelity and safe refusal when sources were weak.
Architecture (shape)
- Ingest — Document AI / OCR + structure extraction for forms and long policies.
- Index — chunking with metadata (doc type, effective date, tenant/scope) into a vector + keyword hybrid store.
- Retrieve — hybrid search with filters; prefer recent / authoritative sources.
- Generate — answer only with retrieved context; surface citations; escalate when confidence is low.
- Controls — PHI-aware handling, access scoping, and audit logs for prompts/retrieval hits.
Approach
- Started with a thin vertical: one high-traffic doc class → end-to-end grounded Q&A.
- Tuned chunking and metadata harder than the model — retrieval quality dominated answer quality.
- Built a small golden set of questions with expected sources for regression before expanding scope.
Trade-offs
- Accepted slower answers when hybrid retrieval + citation checks improved trust.
- Preferred refusal / “ask a human” over fluent hallucination on sparse docs.
- Scoped first release to ops knowledge, not clinical decision support.
Evaluation
- Citation precision: did the answer point at the right source passage?
- Groundedness / contradiction rate on a held-out question set.
- Time-to-answer vs prior Slack/docs search (directional, internal pilot).
Outcome
A production-shaped Document AI + RAG path that made healthcare ops knowledge searchable and citable — the pattern I reuse when regulated teams need GenAI without “chat over a drive folder.”
- Groundedness: answers required retrieval hits; weak-context queries routed to escalation
- Pilot impact (directional): ~30–50% faster lookup for recurring ops questions vs ad-hoc doc search
- Trust: every answer shipped with source links for reviewer verification