Building a Production RAG Assistant with pgvector

Retrieval-Augmented Generation (RAG) is the most reliable way I know to make a large language model answer from your data instead of its imagination. Here is how I built one that ships in production — an in-app assistant that answers from a product’s own documentation, grounded and low on hallucination.

Why RAG instead of fine-tuning

Fine-tuning bakes knowledge into weights; RAG keeps knowledge in a store you can update any time. For a product whose docs change weekly, RAG wins: re-index the changed content and the assistant is current — no training run, no model risk.

The pipeline

Four stages, each independently testable:

1. Chunking. Split source documents by heading into passages small enough to be specific but large enough to carry context. Over-chunking shreds meaning; under-chunking dilutes retrieval.
2. Embeddings. Turn each chunk into a vector with an embedding model (I used Voyage’s lite model). The same model must embed both documents and the user’s question.
3. Vector store. I stored vectors right in PostgreSQL with pgvector and an IVFFlat index, so retrieval lives next to the rest of the app’s data — one database, one backup story.
4. Retrieval & generation. Embed the question, pull the top-k nearest chunks, and pass them to the LLM as grounding with a strict instruction to answer only from the provided context.

Tuning that actually moved the needle

With IVFFlat, the probes setting trades speed for recall. Too low and the assistant misses relevant passages; too high and latency climbs. I tuned probes against a small evaluation set of real questions and measured answer relevance, not just raw vector distance.

The other big lever was idempotent indexing: keying each chunk on a content hash so re-runs only re-embed what changed, and cleaning up orphaned vectors when source docs are deleted. That keeps the index honest and cheap to maintain.

Keeping it grounded

Grounding is a prompt-and-product discipline, not a one-liner. The system prompt tells the model to say “I don’t know” when the context doesn’t cover the question, and the UI surfaces which passages an answer came from. That single habit — show your sources — does more for trust than any clever decoding trick.

Takeaways

• Treat retrieval quality as a first-class metric; evaluate it like you’d evaluate any model.
• Co-locate vectors with your app data if you can — pgvector is plenty for most products.
• Make indexing idempotent from day one; you will re-run it constantly.

RAG isn’t magic, it’s plumbing — good plumbing. Get the four stages right and the assistant feels like it actually read the manual.