Retrieval-augmented generation explained for lawyers
By the ChamberOne team 5 minute read
Vendors of legal AI products often say their tool answers from your documents or from a verified library. The technique behind that claim has a name, retrieval-augmented generation, usually shortened to RAG. You do not need to understand the engineering to evaluate it. You do need to understand what it does, what it cannot do, and which questions expose a weak product. This article explains the idea with an analogy, then turns it into a list you can take into a vendor meeting.
Key points
- RAG retrieves relevant passages first and then asks the model to answer from them, ideally with links.
- It reduces fabrication but retrieval failures, misreading and gaps in the collection remain.
- Insist that every statement links to a source passage and that you read the passage.
- Test with questions that have no answer in the documents, and with conflicting documents.
- Ask vendors about coverage, currency, testing, handling of unknowns and data use.
The problem RAG tries to solve
A language model on its own answers from what it absorbed in training. That knowledge is general, may be out of date and contains nothing about your client's file. When it does not know, it may still produce a confident answer, because producing fluent text is what it does.
For legal work this is a poor fit. You want answers grounded in a specific set of documents: the contract in front of you, the firm's precedents, a library of judgments.
How it works, in plain terms
Picture a research assistant who is not allowed to answer from memory. When asked a question, the assistant first goes to the shelves and pulls out the few pages most likely to be relevant. Then the assistant writes an answer using only those pages, and says which pages were used.
That is the structure of RAG. The collection of documents is cut into passages and indexed. When you ask a question, the system searches for the passages that best match it. Those passages are given to the model along with the question, and the model writes an answer based on them. A well-built product shows you the passages.
Why it reduces fabrication
Giving the model the relevant text and asking it to answer from that text narrows what it needs to invent. If the passage says the notice period is thirty days, the answer is likely to say so. The model has something concrete in front of it, and you have a passage you can check.
Reduced is the right word. It is not eliminated.
- Answers from what it absorbed in training.
- Knowledge is general and may be out of date.
- Knows nothing about your client's file.
- May give a confident answer when it does not know.
- Pulls the most relevant passages first.
- Writes the answer using only those passages.
- Gives the model something concrete to work from.
- Reduces fabrication but does not remove it.
Why it does not remove fabrication
Several things can still go wrong, and each is worth knowing by name.
- Retrieval failure. The search may bring back the wrong passages, or miss the one that matters. A model given the wrong pages will write a fluent answer from them.
- Misreading. The model may summarise a passage inaccurately, drop a qualification or combine two passages that should not be combined.
- Going beyond the text. The model may add statements from its general training that are not in the passages, without marking them.
- Chopping context. A passage separated from its surroundings can mean something different. A clause that applies only in certain circumstances may be retrieved without the condition.
- Gaps in the collection. If the library does not contain the relevant material, the system may produce an answer from whatever is nearest rather than saying nothing.
Questions to put to a vendor
Treat a sales meeting as an examination, and ask for specifics.
- 1
What documents is the tool answering from? Is it my documents, your library or both, and who maintains them?
- 2
Does every statement in an answer link to the passage it came from, so that I can open and read it?
- 3
What does the tool do when it cannot find a relevant passage? Does it say so?
- 4
How do you test accuracy, how often, and can I see results for my kind of work?
- 5
Does the tool ever answer from general knowledge outside the documents, and is that marked?
- 6
How is the collection kept current, and can I see the date of the latest material?
- 7
What happens to my documents and questions? Where are they stored, and are they used for training?
A test you can run yourself
Before buying, give the tool a set of documents you know well and ask it ten questions. Include some with clear answers, some where the answer is not in the documents, and some where two documents conflict. Note whether it finds the right passage, whether it admits when there is no answer and whether it presents a conflict honestly.
A product that answers the unanswerable question with confidence has shown you its weakest point. Better to learn that in a trial than in front of a client.
Suppose a small firm tries a tool on a folder of leases and asks whether any of them allows subletting without consent. The tool names two leases and quotes a clause from each. The partner opens them and finds that one quoted clause is about assignment, not subletting. Retrieval brought back a nearby passage, and the model wrote a confident answer from it. The link made the mistake easy to catch, which is the whole point of insisting on links.
How to use the output
Treat the answer as a pointer to passages, and read the passages. The link to the source is the main safety feature of the whole approach, and it only helps if you click it. For anything that will be relied on, quote from the source and not from the generated summary.
Be especially careful with negatives. A statement that something is not mentioned or not required is hard for a retrieval system to get right, because it depends on everything that was not retrieved.
The same caution applies to questions that begin with every or all. Asking for every indemnity in a hundred contracts requires the search to find all of them, and a retrieval system returns only the passages it ranks highest. Where completeness matters, use the tool to build a first list and then check the documents another way.
- Treat the answer as a pointer to passages.
- Click the link and read the passage.
- Quote from the source, not from the answer.
- Trust a statement that something is not mentioned.
- Assume a search for every or all has found everything.
Confidentiality still applies
A system that answers from your documents must have access to them, and often sends text to an AI provider to produce the answer. Ask where the processing happens and what is kept, and check the guidance from your regulator or bar on confidentiality before you load client material.
This article is general information about the law, legal practice and technology, current when it was published. It is not legal advice. Laws and guidance change: check the sources linked above and take advice on your own circumstances from a lawyer qualified where you are.
Read next
How to check AI legal research before you rely on it
A routine for tracing every AI-suggested authority to its source, confirming it is still good law and testing the proposition it is cited for.
5 minute readGeneral chatbots versus legal research tools: what differs and why it matters
How general chatbots and legal research tools differ on sourcing, links, jurisdiction coverage and currency, and when a general model will do.
5 minute readWriting prompts for legal research questions that return usable answers
A research prompt is a short instruction to a junior. Give it the same detail you would give a person.
4 minute read