Evaluating AI research tools: a checklist for a trial
By the ChamberOne team 4 minute read
Every legal research tool looks good in a demonstration, because the demonstration was chosen to make it look good. The only reliable test is your own questions, in your own jurisdiction, judged by your own lawyers. Librarians, partners and practice managers who run a trial this way learn more in a fortnight than a year of brochures would tell them. This checklist sets out how to design the trial, how to score it and what to keep so that the decision can be defended later.
Key points
- Test with ten real questions from closed files, including a currency question, a local-law question and a false premise.
- Score accuracy, citations, jurisdiction, currency, honesty about limits and speed, and open every citation.
- Keep a separate tally of serious failures such as invented authorities, rather than averaging them away.
- File the evidence and the conditions for use, and repeat a shorter test each year.
Decide what you are testing for
Before you ask a single question, write down what the firm wants from the tool. Is it faster first-pass research for associates? Finding authority in a niche area? Helping non-specialists get oriented? The answer shapes the questions. A tool that is excellent at broad overviews may be poor at pinpoint citations, and the other way round.
Name the people who will judge the output. Give each a short brief and agree who has the final say. A trial judged by whoever happens to be keenest produces a verdict about enthusiasm.
Choose ten real questions
Take ten questions from closed files, ones where you already know a good answer and the authorities that support it. Mix the types so that the trial stresses different features.
- Two or three questions on settled law, where the answer is clear and well documented.
- Two on a recent change in legislation or case law, to test currency.
- Two on your own jurisdiction's specifics, such as local procedure, where foreign material can mislead.
- One where the right answer is that the law is unclear or there is no direct authority.
- One that contains a false premise, to see whether the tool challenges it or goes along.
- One long, fact-heavy question, to test how it handles detail.
Run the same questions under the same conditions
If you are comparing tools, ask each the identical question, worded the same way, on the same day, by the same person. Record the exact prompt and the full output, including the links offered. Screenshots are fine, but saved text is better, because you will want to search it later.
Run a few questions twice, on different days. Some tools give different answers to the same question, and that variability is itself a finding.
What to score
Use a simple scale, say one to five, on each of a handful of criteria. Keep the criteria few so scorers apply them consistently.
- Accuracy: does the answer state the law correctly for the date and place asked about?
- Citations: do the authorities exist, and does each say what the tool claims? Open every one.
- Jurisdiction: does the tool stay in the right legal system, or mix in material from elsewhere?
- Currency: does it reflect recent changes, and does it say which version it relies on?
- Honesty about limits: does it flag uncertainty, gaps and conflicting authority?
- Speed and usability: how long did it take, and how easy was it to check the sources?
Count the failures, do not average them away
An average score hides the thing that matters most: whether the tool ever produced a confident, fabricated answer. Keep a separate tally of serious failures, such as an invented authority, a repealed provision presented as current or a quotation that does not appear in the source. One of these in ten questions is a fact about the tool that no average should soften.
Note also the failures a lawyer would have caught and those they would not. A wrong answer that looks wrong is less dangerous than one that looks right.
Look beyond the answers
The research output is only part of the evaluation. Ask the supplier the practical questions: where does the text of your queries go, and is it kept? What sources does the tool search, and how often are they updated? Does it cover your jurisdiction and the courts and legislation you rely on? Can you see the source behind each statement? What does it cost at the usage you expect, and what happens to your data if you leave?
Check access and logs too. Can the tool record who asked what? Can you switch off features that send client information? A research tool that works but cannot be controlled may not be one you can responsibly adopt.
Keep the evidence
File the questions, the outputs, the scores, the names of scorers and the date. Add a one-page summary of the decision and the reasons. That record serves several purposes. It supports the purchase to the partners, answers a client who asks how you assessed the tool, and gives you a baseline for re-testing after the supplier changes its models or sources.
Plan to repeat a shortened version of the test each year, or when the supplier announces a major change. A tool that scored well last year is not guaranteed to score well on its new model.
A fair verdict
Resist the urge to look for a winner at all costs. The right answer may be that a tool is useful for orientation and for locating candidate authorities, and unsuitable for anything a lawyer would rely on without opening the source. That is a perfectly good result, and it tells people exactly how to use it.
Write down the conditions for use: which tasks, which checks and which people. Without that, a favourable trial turns into a general licence to trust the tool, and that is a different decision from the one you tested.
Share the result with the people who will use the tool, including the sceptics, and invite them to try to break it. Their attempts often find failure modes that the structured test missed.
- Keep a separate tally of serious failures
- Write down the conditions for use
- Share the result with users, including sceptics
- Average serious failures away
- Look for a winner at all costs
This article is general information about the law, legal practice and technology, current when it was published. It is not legal advice. Laws and guidance change: check the sources linked above and take advice on your own circumstances from a lawyer qualified where you are.
Read next
How to check AI legal research before you rely on it
A routine for tracing every AI-suggested authority to its source, confirming it is still good law and testing the proposition it is cited for.
5 minute readGeneral chatbots versus legal research tools: what differs and why it matters
How general chatbots and legal research tools differ on sourcing, links, jurisdiction coverage and currency, and when a general model will do.
5 minute readRetrieval-augmented generation explained for lawyers
A plain explanation of how AI answers from a document set, why it reduces but does not remove fabricated content, and what to ask vendors.
5 minute read