The fact that language models "hallucinate" is a consequence of the way language models work. To reduce the effect, Retrieval-Augmented Generation can help to visualize the answer quality and the resilience of the answer. Retrieval Augmented Generation (RAG) has evolved from an academic concept to one of the most widely used patterns in production AI applications. In this Executive Breakfast, we used a legal tech project to show how RAG pipelines can be built to be robust, maintainable and secure.
The problem with pure LLM answers
Large language models have a fundamental problem for business-critical applications: their knowledge is limited to the training data set and is outdated. For a law firm that wants to query current case law and internal documents, a model whose knowledge ends in 2023 is simply not sufficient.
RAG solves this problem by combining the language model with a retrieval system. Before each answer is generated, relevant documents are retrieved from a vector database and given to the model as context. The model generates an answer based on real, up-to-date documents - and can cite sources.
Architecture of the RAG pipeline
The first step is indexing: documents are split into chunks, converted into embeddings and stored in pgvector. The quality of the chunks is crucial - too large and the context becomes unspecific; too small and important connections are lost.
At query time, the user question is also converted into an embedding and semantically similar chunks are retrieved from the database. These are passed together with the original question to the language model, which generates a precise, source-based answer.
Authentication and data access
For a law firm, data separation is not an optional extra - it is a legal necessity. Keycloak controls access to documents based on client roles. Each query is carried out with the token of the requesting user so that only authorized documents are included in the context.
Quality assurance and evaluation
Without systematic evaluation, a RAG pipeline is a black box system. We implemented an automated test framework that executes a sample of queries daily and evaluates the response quality based on defined criteria. Regression was thus detected at an early stage.
The result: a production-ready AI application that lawyers trust - because they can see and check the sources.