How I Built a Document-Grounded Question Answering Flow

The core idea behind this system is straightforward: when a user asks a question, the application should not send that question directly to an LLM and hope for the best. Instead, it should authenticate the user, find the most relevant pieces of the user's own documents, build a grounded prompt, stream the answer back to the frontend, and keep enough records to debug or improve the system later.

That creates a retrieval-augmented generation flow, often called RAG. The LLM is still responsible for language generation, but the application is responsible for access control, retrieval, prompt construction, streaming, and persistence.

1. The User Asks a Question

The flow begins in the chat interface. A user selects the documents they want to use as context, types a question, and submits it.

The frontend sends two important pieces of information to the API:

  • the user's question
  • the selected document IDs

Those document IDs are not just UI state. They become part of the retrieval boundary. The user is not asking across every document in the database; they are asking across the documents they intentionally selected.

The frontend also creates an optimistic user message and an empty assistant message. That makes the chat feel immediate while the backend does the heavier work of embedding, retrieval, LLM generation, and persistence.

2. The API Authenticates and Authorizes Access

Before doing any retrieval or generation, the API validates the user's session.

Authentication answers the question: "Who is making this request?"

Authorization answers the next question: "Is this user allowed to access the requested conversation and documents?"

The API uses the authenticated user ID to scope all downstream work. Conversations are loaded only for that user. Selected files are checked against that user. Retrieved chunks come only from documents the user is allowed to use.

This matters because RAG systems can accidentally become data-leak systems if retrieval is not scoped correctly. The vector search layer needs the same access-control discipline as any other database query.

3. The Query Is Embedded

Once the user and request are valid, the system converts the question into an embedding.

An embedding is a numeric representation of the meaning of the text. The documents have already been split into chunks, and each chunk has its own stored embedding. By embedding the user's question into the same vector space, the system can compare the question to document chunks by semantic similarity.

In this implementation, the query embedding is created through an embeddings model and returned as a float vector. The stored chunk embeddings use the same dimensionality, which allows the database to compare them directly.

This step turns the user's natural-language question into something the retriever can use.

4. The Retriever Searches pgvector with Metadata Filters

The retriever searches document chunks stored in PostgreSQL with pgvector.

The search has two parts:

  • metadata filtering
  • vector similarity ranking

The metadata filters narrow the candidate set before ranking. For example, the retriever filters by selected document ID so the answer is grounded only in the files attached to the conversation. It can also filter by processing status, chunk index, or a set of allowed file IDs.

After filtering, pgvector ranks the remaining chunks by distance from the query embedding. In this flow, cosine distance is used to find the chunks whose meaning is closest to the user's question.

This gives the system a small, focused set of document excerpts instead of sending entire files to the LLM.

5. An Optional Reranker Can Improve the Result Set

The first retrieval pass is fast and efficient, but it is not always perfect. Vector similarity is good at narrowing the search space, but a second ranking step can often improve answer quality.

A reranker can take the top candidate chunks and reorder them using a more expensive but more precise relevance model. That could be a cross-encoder, an LLM scoring pass, or another ranking model.

The clean place for reranking is after vector retrieval and before prompt assembly:

  1. retrieve the top candidate chunks from pgvector
  2. score or reorder those chunks with the reranker
  3. keep the best final chunks for the prompt

This makes reranking optional. The system still works with pgvector alone, but it has a clear path to better relevance when needed.

6. The Prompt Builder Assembles the Question, Chunks, and Citation Metadata

After retrieval, the system builds the prompt that will be sent to the LLM.

The prompt includes:

  • the user's original question
  • the retrieved document chunks
  • metadata about where those chunks came from
  • system instructions that tell the model to answer only from the provided context

This is where the application turns retrieval results into usable model context. The prompt builder groups chunks by source document and includes labels such as filenames or document IDs. That gives the model enough context to produce grounded answers and, when available, cite the source material.

The system instructions are just as important as the chunks. They tell the assistant to stay within the retrieved context, avoid unsupported claims, and be clear when the provided documents do not contain enough information.

The goal is not to make the LLM "know" the documents permanently. The goal is to give it the right temporary context for this one answer.

7. The LLM Generates the Answer

Once the prompt is assembled, the API sends the request to the LLM.

The model receives the user question, the retrieved context, and the grounding instructions. It then generates an answer based on the provided chunks rather than relying on general knowledge.

Conversation metadata is also attached to the LLM request. That metadata makes it easier to trace a model response back to the user, conversation, and application request that produced it.

The important design choice is that the LLM is not responsible for deciding which private documents it should see. The application decides that through authentication, authorization, and retrieval. The LLM only receives the context that has already passed those checks.

8. The API Streams the Answer to the Frontend

Instead of waiting for the entire answer to finish, the API streams the response back to the frontend as it is generated.

The backend receives streamed events from the LLM provider, normalizes them into application-level events, and forwards text deltas to the browser using server-sent events.

The frontend listens for events such as:

  • new conversation created
  • user message persisted
  • assistant text delta received
  • final assistant message persisted
  • stream complete
  • error

As each text delta arrives, the frontend appends it to the optimistic assistant message. The user sees the answer appear progressively instead of waiting on a blank screen.

When generation completes, the backend stores the final assistant message and sends that persisted message back to the frontend. The UI then replaces the optimistic placeholder with the saved record.

9. Query, Response, and Retrieval Logs Are Stored in Postgres

The final part of the flow is persistence.

At minimum, the system stores:

  • the conversation
  • the user's message
  • the assistant's final response
  • the selected document IDs used as context
  • the LLM response ID, when available

The data model also supports deeper observability through query and retrieval logs. Those logs are designed to capture things like:

  • original query text
  • normalized query text
  • model name
  • prompt version
  • response text
  • token counts
  • latency
  • retrieval scores
  • retrieval ranks
  • retrieval strategy
  • metadata filters used during search

This logging layer matters because RAG quality is hard to improve without traces. If an answer is wrong, you need to know whether the issue came from bad retrieval, missing chunks, weak ranking, prompt construction, or model behavior.

Persisting query and retrieval details makes the system inspectable instead of mysterious.

The Full Flow

The end-to-end flow looks like this:

  1. A user asks a question in the chat UI.
  2. The API authenticates the user and scopes access to that user's documents.
  3. The question is converted into an embedding.
  4. The retriever searches pgvector using document filters and vector similarity.
  5. An optional reranker can reorder the retrieved candidates.
  6. The prompt builder combines the question, chunks, metadata, and grounding instructions.
  7. The LLM generates an answer from the provided context.
  8. The API streams answer deltas back to the frontend.
  9. The system stores the conversation, response, retrieval details, and logs in Postgres.

The main lesson is that a good document Q&A flow is more than an LLM call. The quality and safety of the answer come from the whole pipeline: authentication, scoped retrieval, good chunk selection, clear prompt construction, streaming UX, and durable logs.