ChamberOne

Evaluating AI research tools: a checklist for a trial

By the ChamberOne team 4 minute read

Every legal research tool looks good in a demonstration, because the demonstration was chosen to make it look good. The only reliable test is your own questions, in your own jurisdiction, judged by your own lawyers. Librarians, partners and practice managers who run a trial this way learn more in a fortnight than a year of brochures would tell them. This checklist sets out how to design the trial, how to score it and what to keep so that the decision can be defended later.

Key points

  • Test with ten real questions from closed files, including a currency question, a local-law question and a false premise.
  • Score accuracy, citations, jurisdiction, currency, honesty about limits and speed, and open every citation.
  • Keep a separate tally of serious failures such as invented authorities, rather than averaging them away.
  • File the evidence and the conditions for use, and repeat a shorter test each year.

Decide what you are testing for

Before you ask a single question, write down what the firm wants from the tool. Is it faster first-pass research for associates? Finding authority in a niche area? Helping non-specialists get oriented? The answer shapes the questions. A tool that is excellent at broad overviews may be poor at pinpoint citations, and the other way round.

Name the people who will judge the output. Give each a short brief and agree who has the final say. A trial judged by whoever happens to be keenest produces a verdict about enthusiasm.

Choose ten real questions

Take ten questions from closed files, ones where you already know a good answer and the authorities that support it. Mix the types so that the trial stresses different features.

  • Two or three questions on settled law, where the answer is clear and well documented.
  • Two on a recent change in legislation or case law, to test currency.
  • Two on your own jurisdiction's specifics, such as local procedure, where foreign material can mislead.
  • One where the right answer is that the law is unclear or there is no direct authority.
  • One that contains a false premise, to see whether the tool challenges it or goes along.
  • One long, fact-heavy question, to test how it handles detail.

Run the same questions under the same conditions

If you are comparing tools, ask each the identical question, worded the same way, on the same day, by the same person. Record the exact prompt and the full output, including the links offered. Screenshots are fine, but saved text is better, because you will want to search it later.

Run a few questions twice, on different days. Some tools give different answers to the same question, and that variability is itself a finding.

What to score

Use a simple scale, say one to five, on each of a handful of criteria. Keep the criteria few so scorers apply them consistently.

  • Accuracy: does the answer state the law correctly for the date and place asked about?
  • Citations: do the authorities exist, and does each say what the tool claims? Open every one.
  • Jurisdiction: does the tool stay in the right legal system, or mix in material from elsewhere?
  • Currency: does it reflect recent changes, and does it say which version it relies on?
  • Honesty about limits: does it flag uncertainty, gaps and conflicting authority?
  • Speed and usability: how long did it take, and how easy was it to check the sources?

Count the failures, do not average them away

An average score hides the thing that matters most: whether the tool ever produced a confident, fabricated answer. Keep a separate tally of serious failures, such as an invented authority, a repealed provision presented as current or a quotation that does not appear in the source. One of these in ten questions is a fact about the tool that no average should soften.

Note also the failures a lawyer would have caught and those they would not. A wrong answer that looks wrong is less dangerous than one that looks right.

Look beyond the answers

The research output is only part of the evaluation. Ask the supplier the practical questions: where does the text of your queries go, and is it kept? What sources does the tool search, and how often are they updated? Does it cover your jurisdiction and the courts and legislation you rely on? Can you see the source behind each statement? What does it cost at the usage you expect, and what happens to your data if you leave?

Check access and logs too. Can the tool record who asked what? Can you switch off features that send client information? A research tool that works but cannot be controlled may not be one you can responsibly adopt.

Keep the evidence

File the questions, the outputs, the scores, the names of scorers and the date. Add a one-page summary of the decision and the reasons. That record serves several purposes. It supports the purchase to the partners, answers a client who asks how you assessed the tool, and gives you a baseline for re-testing after the supplier changes its models or sources.

Plan to repeat a shortened version of the test each year, or when the supplier announces a major change. A tool that scored well last year is not guaranteed to score well on its new model.

A fair verdict

Resist the urge to look for a winner at all costs. The right answer may be that a tool is useful for orientation and for locating candidate authorities, and unsuitable for anything a lawyer would rely on without opening the source. That is a perfectly good result, and it tells people exactly how to use it.

Write down the conditions for use: which tasks, which checks and which people. Without that, a favourable trial turns into a general licence to trust the tool, and that is a different decision from the one you tested.

Share the result with the people who will use the tool, including the sceptics, and invite them to try to break it. Their attempts often find failure modes that the structured test missed.

Do
  • Keep a separate tally of serious failures
  • Write down the conditions for use
  • Share the result with users, including sceptics
Don't
  • Average serious failures away
  • Look for a winner at all costs
Reaching a fair verdict on a trial

This article is general information about the law, legal practice and technology, current when it was published. It is not legal advice. Laws and guidance change: check the sources linked above and take advice on your own circumstances from a lawyer qualified where you are.

Read next

Evaluate a legal AI research tool: a trial checklist