Hallucinated citations: what court sanctions teach about checking AI output
By the ChamberOne team 5 minute read
Courts in several countries have now dealt with filings that cited cases which do not exist or that attributed to real cases things they never said. The details differ, and this article does not retell individual cases. What is more useful to a litigator is the pattern: how these errors get into a document, why nobody stopped them, and which small controls would have. None of the controls is technical.
Key points
- The common failure is a skipped check, so make verification a named step with a named person.
- A verification note attached to each court document stops responsibility from diffusing.
- Check the proposition and any quotation, not just that the case exists.
- Never ask the tool that produced a citation to confirm it. Use a primary source.
- If an error is found after filing, tell your supervising lawyer immediately and check your duties to the court.
Why this keeps happening
General-purpose language models generate text that is likely to follow from the prompt. A citation has a familiar shape: a party name, a year, a court, a number. A model can produce something with exactly that shape without any underlying case. The result reads as authoritative because it is formatted like authority.
Regulators have noticed. The Solicitors Regulation Authority in England and Wales, in a published report on AI in the legal market, describes AI-drafted legal arguments that included non-existent cases, and notes that people may place more trust in computers than in humans.
The first pattern: the checking step was skipped, not failed
In most reported incidents, the problem was not that a check was done badly. It was that nobody did one. A lawyer or litigant took text from a tool, saw that it read well, and put it into a document that went to the court.
The fix is a rule that nothing is cited until a named person has opened the source and read the relevant passage. That is a thirty-second step per authority.
The second pattern: the work passed through several hands
A junior drafts, a colleague tidies the formatting, a partner signs. Each assumes an earlier person verified the authorities. Responsibility diffuses until nobody holds it.
A practical control is to attach a verification note to every court document. It lists the authorities, the person who checked each, the date, and the paragraph relied on. The signing lawyer reads the note before signing. If there is no note, the document does not go out.
- Each person assumes an earlier person checked.
- Responsibility spreads until nobody holds it.
- A missing check is invisible at signing.
- It lists each authority, who checked it, the date and the paragraph.
- The signing lawyer reads it before signing.
- No note, no filing.
Two quieter patterns: wrong propositions and self-confirmation
Not every error involves an invented case. Some documents cite genuine decisions for points those decisions do not support. A check that stops at confirming the case exists will miss this.
The control here is to check the proposition. The person verifying should be able to say which paragraph supports the sentence in the draft and, if there is a quotation, to find the exact words.
Several reported episodes involve the same sequence. A problem is raised by the court or the other side. The lawyer asks the same tool whether the authority is real, and the tool says it is. A tool cannot reliably verify its own output.
Make it a rule that verification is done against a primary source or an established legal database, never by asking the generating tool to confirm itself.
- Say which paragraph supports each sentence in the draft.
- Find the exact words of any quotation.
- Verify against a primary source or an established legal database.
- Stop at confirming that the case exists.
- Ask the tool that produced a citation to confirm it.
Controls a supervising partner can put in place this week
- A one-page policy saying who may use AI tools for research and drafting, for what, and with what review.
- A verification note for every filing, submission and skeleton argument, signed by the person who checked.
- A rule that unfamiliar authorities, and any authority supplied by a client or a litigant in person, are verified before they are relied on or responded to.
- A requirement that juniors say when AI was used in a draft, so the reviewer knows to look harder.
- A short briefing for support staff and trainees, since errors can enter at any point.
- A named person to ask when in doubt, and a firm culture where asking is rewarded.
When the error is found, and what it says about AI use
Courts have generally responded better to prompt, candid correction than to explanation after the fact. If you discover that a document you filed contains an authority you cannot verify, tell your supervising lawyer at once, then consider with them what the court and the other side should be told and how quickly.
Check the guidance from your regulator or bar on duties to the court when an error is discovered. Also consider your professional indemnity insurer's notification requirements.
None of this means AI should be kept away from litigation. Tools can help with organising documents, preparing summaries and drafting first versions of material that a lawyer then owns. The problem is the unverified step between a tool's output and a court. Closing that step is a matter of process and supervision, and a small firm can do it as well as a large one.
A hypothetical afternoon, and fakes from the other side
Suppose a three-lawyer litigation firm has a response due on Friday. The trainee drafts with the help of a tool and includes six authorities. The associate reads the draft for style and sends it to the partner. The partner, who is in court all week, signs it on Thursday night because two people have already looked at it.
Nobody in that chain was careless in the ordinary sense. Each did a reasonable job on the part they were asked to do. The gap was that no one was asked to check the authorities, and the draft gave no sign that this was needed.
Now add a verification note, a one-page table that the trainee fills in as part of the draft. Two of the six rows are empty on Thursday, and the partner sees it immediately. The error is caught because the process makes a missing check visible, not because anyone became more cautious.
Fake authorities do not only arrive from your own team. A litigant in person, or an opposing party, may cite material that came from a chatbot. Build the habit of checking authorities cited against you before you respond to them. If you cannot find one, say so politely and ask for a copy.
Where you are appearing before a judge who may rely on both sides' citations, it is also in your interest, and your duty as an officer of the court, to be open about difficulties. Check the guidance from your regulator or bar for what that means where you practise.
Sources
This article is general information about the law, legal practice and technology, current when it was published. It is not legal advice. Laws and guidance change: check the sources linked above and take advice on your own circumstances from a lawyer qualified where you are.
Read next
AI in Witness Bundle and Proof of Evidence Preparation: What to Automate and What Not To
A practical guide for litigators on which parts of witness bundle and proof of evidence preparation AI can handle, and which parts it must not touch.
6 minute readBuilding a litigation chronology with AI
Extract dated events, link each to its source page and check for conflicts before the chronology is used.
5 minute readSummarising witness statements with AI and spotting inconsistencies
AI can line up statements by topic, but cross-examination planning must rest on the original words.
4 minute read