General chatbots versus legal research tools: what differs and why it matters
By the ChamberOne team 5 minute read
Two lawyers ask the same question about a limitation period. One uses a general chatbot and gets a fluent paragraph. The other uses a tool built for legal research and gets a shorter answer with links to a statute and two judgments. The first answer may well be right. The difference is that only the second gives its author something to check. This article compares the two kinds of tool on the points that matter in practice and says when each is acceptable.
Key points
- A general chatbot reconstructs a plausible answer. A legal research tool retrieves sources and links to them.
- Compare sourcing, citation links, jurisdiction coverage, currency and treatment of later history.
- Test with questions from your own jurisdiction to which you already know the answers.
- A general model suits explaining concepts and structuring notes, never supplying authority.
- A research tool reduces the checking burden but does not remove it.
What a general chatbot is doing
A general-purpose chatbot is a language model trained on a very large body of text. When you ask a question it produces the answer that is most plausible given that training. It does not look up the law in the way a database does. It reconstructs a likely answer, and where its training is thin, it may reconstruct something that sounds right and is not.
Some chatbots can now browse the web or search documents you provide. That helps, but the answer is still a generated summary, and what it searched and how it chose may not be visible to you.
What a legal research tool is trying to do
A tool built for legal research normally sits on top of a collection of legal sources, such as legislation, judgments and commentary. A typical design retrieves passages from that collection and asks a model to answer using them, with references back to the source. The model's text is only part of the product. The collection and the links are the rest.
This does not make such a tool infallible. It can retrieve the wrong passage, misread it, or summarise it badly. But it gives you a place to click.
- Produces the answer most plausible from its training.
- Does not look up the law the way a database does.
- Browsing still gives a generated summary.
- Sits on a collection of legal sources.
- Retrieves passages and answers using them, with references.
- Can still retrieve or misread the wrong passage.
- Gives you a place to click.
Five points of comparison
Whatever the marketing says, test each of these before you pay for a tool or rely on one.
- 1
Sourcing. Where does the answer come from? Can the tool show you the passage, and is it from an official or recognised source?
- 2
Citation links. Does each authority link to the underlying document, so you can read it, or only name it?
- 3
Jurisdiction coverage. Does the tool cover the courts and legislation you work in, including your country's own materials and any local reporting series?
- 4
Currency. How often is the collection updated, and does the tool tell you the date of the latest material it holds?
- 5
Treatment. Does it tell you whether a case has been followed, distinguished or overruled, or leave that to you?
Why jurisdiction matters more than people expect
Many tools perform best on the law of the largest markets, where the volume of published material is highest. A lawyer in a smaller jurisdiction may find that the tool has less of their law, or mixes in decisions from other countries and presents them as if they applied.
Test with questions to which you already know the answer, drawn from your own practice. Ask about a recent local statute, a leading local case and a procedural rule. If the tool is vague, wrong or silent on these, you have learned what you need to know.
Include one trick question: a point on which the law changed recently, or on which two jurisdictions differ. A good tool flags the uncertainty. A poor one answers confidently from the wrong source.
Ask vendors direct questions
- Which sources do you hold for my jurisdiction, and how do you obtain them?
- How often is the collection updated, and can I see the date of the most recent item?
- Does every statement in an answer link to a source passage?
- What does the tool do when it cannot find an answer?
- How do you test for fabricated or misdescribed authorities, and can you show results?
- What happens to the questions and documents I enter?
When a general model is still acceptable
There are uses where a general chatbot is a reasonable choice, provided no confidential material goes in and nothing unverified comes out. Explaining a concept in plain language before you start research. Suggesting search terms or alternative ways to frame an issue. Drafting a structure for a note that you will fill in yourself. Rewriting your own text to make it clearer.
In each of these you are using the tool to think or write, not to supply authority. If the output ends up in advice or a filing, every proposition should be traced to a primary source by you.
When it is not
Avoid a general chatbot as the source of authority for anything that will be cited to a court, relied on in advice, or used in a transaction. Avoid it for questions of current law in a small or fast-changing area. Avoid it for any task involving client confidential material unless you have checked the retention and training terms.
And do not treat a legal research tool as a licence to stop checking. The checking is lighter, because the links are there. It is still required.
A simple procurement test
Before buying, take ten questions from your own recent files. Give them to the tool and to a colleague. Compare the answers, check every citation, and note where the tool failed and how it failed. A tool that fails visibly, by saying it cannot find authority, is safer than one that fails confidently.
Repeat the test every few months. Tools change, and so does the law.
Suppose a four-lawyer commercial firm runs this test on two tools. One gives longer, smoother answers and cites eight authorities for a point on which the firm's own partner would cite two. Only five of those eight can be found. The other tool gives terse answers, links each to a source and says plainly that it found nothing on one question. The second is less impressive in a demonstration and much more useful on a Tuesday afternoon.
This article is general information about the law, legal practice and technology, current when it was published. It is not legal advice. Laws and guidance change: check the sources linked above and take advice on your own circumstances from a lawyer qualified where you are.
Read next
How to check AI legal research before you rely on it
A routine for tracing every AI-suggested authority to its source, confirming it is still good law and testing the proposition it is cited for.
5 minute readRetrieval-augmented generation explained for lawyers
A plain explanation of how AI answers from a document set, why it reduces but does not remove fabricated content, and what to ask vendors.
5 minute readWriting prompts for legal research questions that return usable answers
A research prompt is a short instruction to a junior. Give it the same detail you would give a person.
4 minute read