ChamberOne

Learning to spot AI errors: practice exercises for legal teams

By the ChamberOne team 5 minute read

Telling people to check AI output rarely works. They agree, nod and then trust a smooth paragraph because it reads well and the deadline is close. Checking is a skill, and skills improve with practice. This article is for training partners and team leaders who want a set of short exercises, each built around a planted error, that make checking a habit. You can run each drill in fifteen minutes at a team meeting, using material from your own practice.

Key points

  • A planted error in a safe setting teaches checking better than a warning does.
  • Use public or invented material, never real client text, and run drills of about fifteen minutes.
  • Teach the order of checks: confirm the source exists in a trusted place, then confirm it says what is claimed.
  • Make the debrief the focus, and turn real near-misses into new drills.

Why drills work better than warnings

Reading a warning that AI can be wrong does not change behaviour. Being caught out does. A person who has once accepted a confident but false paragraph in a safe setting, and been shown the error, will remember it. That is the idea behind a drill: a low-stakes failure that leaves a memory.

Drills also give a team a shared language. When everyone has seen a planted wrong date or an invented authority, 'I think this is one of those' becomes something people feel able to say to a senior colleague.

How to build an exercise

Take a short piece of text from your own field: a paragraph of advice, a clause summary or a research note. Plant between one and three errors. Give it to the team as if it came from an AI tool, with the source documents to hand, and a time limit. Ask them to mark what is wrong and say how they found out.

Do not use real client material. Use public documents or invented facts, and say clearly that the exercise text is made up. Where you need an authority, use one you have checked yourself, or use a plainly fictional one that the group knows is fictional, so nobody mistakes it for law.

Drill one: the invented authority

Write a paragraph of research that cites three authorities. Two are real and you have read them. The third is invented, with a plausible name, a plausible court and a neat proposition. Ask the team to confirm each authority exists and says what the paragraph claims.

The lesson is the order of checking. The team should look the authority up in a trusted database first, and only then ask whether it supports the point. Many people check the proposition by asking the same tool again, which proves nothing. Debrief on where each person looked.

Drill two: the wrong date and the wrong number

Give a summary of a contract, with the contract itself. Change a notice period, swap two dates and alter one sum. Keep the summary fluent. Ask the team to find the differences within ten minutes.

Most groups find the sum and miss the notice period. That is the point. Numbers and dates are where fluent prose lets the eye slide past. After the drill, agree a rule: every number and date in a summary is compared against the source, every time.

Drill three: the subtle misquote

Take a short passage from a public judgment or a piece of legislation you have verified. Present a 'quotation' of it in which one word has changed, such as 'may' for 'must', or in which a qualifying phrase has been dropped. Ask the team to compare it word by word.

This drill teaches that quotation marks around AI text do not mean the words are exact. Models paraphrase and then present the paraphrase as a quote. The habit to build is to copy quotations from the source, not from the tool, and to compare any quote by pasting the two versions side by side.

Drill four: the confident answer to the wrong question

Ask a question that depends on jurisdiction or date, such as a limitation question, and supply an answer that is correct for a different jurisdiction or the law as it stood years ago. Ask the team what the answer assumes.

Here the error is not in any single sentence. The whole answer fits a different case. The skill is to ask of every output: for which place, which date, which facts. Another drill in the same family uses an answer that ignores an important fact in the instructions.

A related exercise uses an answer that ignores a fact in the instructions, such as a client's residence, the value of a claim or a date of signing. Ask the team to list the facts the answer relied on and the facts it left out. People who do this once tend to read the instructions again before they read the output.

Running the programme and keeping it going

Run one drill a month, rotate who sets it, and keep a simple score of what the group caught. Do not turn it into a test that embarrasses people. Make the debrief the main event: what clue should have prompted a check, and what is the quickest reliable way to check this kind of error?

Keep a one-page checklist from each session, covering the clue, the check and the place to look, and pin it where the team works.

Collect real examples from your own use of AI tools, with client details removed, and turn them into new drills. Add a short section on checking to your onboarding. Encourage people to report near-misses without blame, because a team that hides errors cannot learn from them.

Over time the drills shift habits. People start to ask where a claim came from before they ask whether it reads well. That small change in reflex is worth more than any policy document.

A useful variation is to hand the same text to two people in different roles, say a trainee and a senior associate, and compare what each found and how long it took. Seniors often catch legal problems and miss typing slips, while trainees do the reverse, which shows why review by two people beats review by one.

Keep a log of the drills, the errors planted and the share the group caught, so you can see whether the habit is taking hold. If the group catches everything, make the next errors subtler. If it catches little, slow down and repeat an earlier drill.

  1. 1Run one drill a month
  2. 2Debrief on what the group caught
  3. 3Pin a one-page checklist
  4. 4Turn real near-misses into the next drill
  5. Then back to step 1.
A monthly drill routine

This article is general information about the law, legal practice and technology, current when it was published. It is not legal advice. Laws and guidance change: check the sources linked above and take advice on your own circumstances from a lawyer qualified where you are.

Read next

Training lawyers to spot AI errors: practice drills