The duty to supervise AI outputs
By the ChamberOne team 5 minute read
Every partner knows how to supervise a trainee. You explain the task, set a deadline, ask to see the work and read it before it goes out. The same discipline carries over to AI tools almost unchanged, with one difference that makes it more important: the tool does not hesitate. A trainee who is unsure will often say so. A tool delivers a doubtful answer in the same tone as a certain one.
Key points
- Supervise AI as you would a fast, willing trainee: brief, check and take responsibility.
- Give narrow instructions and supply the source material.
- Set tiers of review so that the check matches the risk, with full verification for filings.
- Record on the file which tool was used and who reviewed what.
- Give supervisors time, and look for signs of rubber-stamping.
The trainee comparison, and where it breaks down
The comparison is useful because it brings familiar habits to mind. A trainee is quick, willing and sometimes wrong. They need a clear brief, they should not work unseen on anything important, and their work is the supervisor's responsibility once it leaves the room. All of that applies to an AI tool.
Where the comparison fails is just as important. A trainee learns from your corrections; a tool generally does not, unless it is configured to. A trainee knows when something feels off; a tool has no such sense. A trainee will tell you if they have used a shortcut; a tool will not. So supervision of a tool has to rely on process rather than on the tool's own honesty.
- Learns from your corrections
- Knows when something feels off
- Tells you if they used a shortcut
- Generally does not learn unless configured to
- Has no sense that something is off
- Will not tell you about a shortcut
Define the task before you start
Supervision begins with a good instruction. Vague instructions produce unpredictable output, and unpredictable output is difficult to check. Decide what you want the tool to do, what material it should use, and what form the result should take.
- State the task narrowly: summarise this letter, list the dates in this agreement, draft a first version of this notice.
- Provide the source material rather than asking the tool to recall it.
- Say what the tool must not do, such as adding advice or changing figures.
- Ask for output that is easy to verify, with references to where each point came from.
- State the task narrowly
- Provide the source material
- Say what the tool must not do
- Ask for references to where each point came from
- Give vague instructions
- Ask the tool to recall the source material
Decide in advance what the check will be
The strength of the check should match the risk. A summary for internal use needs less scrutiny than a submission to a court. Write down the tiers so that people do not decide ad hoc, under pressure.
For low-risk uses, such as an internal note, a quick read for sense may be enough. For medium-risk uses, such as a client email or a first draft of a letter, a line-by-line read against the source. For high-risk uses, such as anything filed with a court or sent to a regulator, every fact, quotation and authority is opened and verified by a person, and a second person signs off.
What a proper review looks like
A review is more than reading for flow. A fluent paragraph can be wrong in a single word. Teach people to review in layers: first, does the output answer the question that was asked? Second, are the facts consistent with the source? Third, are the legal statements correct for this jurisdiction and still current? Fourth, is anything missing that should be there?
Missing material is the hardest to spot. A summary that omits an unfavourable clause looks complete. Reviewers should know the document well enough to ask what a good version would contain, and then check for it.
Authorities deserve particular attention. Open each one, read the relevant passage, and confirm that it says what the draft claims. Do not accept a citation because the format looks right.
Who is accountable, and how to show it
The person who signs or sends the document is accountable for it. So is the supervisor, where one is responsible for the work. This does not change because a tool produced the first draft. Check the guidance from your regulator or bar on supervision and technology, because the details differ across jurisdictions.
Make accountability visible. Keep a note on the file of which tool was used, for what, who reviewed the result and when. It need not be long. A line such as 'first draft produced with the firm's approved tool; reviewed line by line against the lease by the associate; authorities checked by the supervising partner' is enough, and it helps if a question arises later.
Supervision at scale: policy and training
In a larger firm, supervision cannot depend on individual habits. A short policy should name the approved tools, the permitted uses, the review standard for each risk tier and who to ask. Training should include examples of failure, so that people see what an invented authority or a quietly altered figure looks like.
Supervisors also need time. If a partner is asked to review work from a tool in addition to everything else, without any adjustment to their workload, reviews will get thinner. Build review time into estimates and billing, and watch for signs of rubber-stamping.
A scenario: the associate and the lease summary
Suppose an associate in a small property firm uses a tool to summarise a long commercial lease for a client. The summary is tidy and says the tenant may assign with consent. The partner, short of time, reads only the summary. The lease, in a schedule, says consent may be withheld absolutely in the first five years. The summary was fluent, plausible and wrong about the very point the client cared about.
A supervision routine would have caught it. The associate would have been asked to give page references for each point, the partner would have spot-checked the assignment clause, and the file note would show who had read the lease itself. None of this is elaborate. It is the supervision a partner would give a trainee, applied without exception.
Signs that supervision is failing
Look for patterns. Documents that arrive with identical phrasing across different matters. Authorities that nobody can produce on request. Reviewers who return work within minutes. Staff who cannot say which tool was used. Any of these suggests the checks are not happening, and should prompt a conversation rather than a reprimand.
- Spot-check a sample of recent documents each month by tracing every citation.
- Ask staff to explain a conclusion in a document without looking at it.
- Review any near-miss, even if no harm resulted.
AI drafts and an audit record in ChamberOne
- Produces matter briefs, document analyses, suggested tasks and Ask AI answers as drafts.
- Keeps each output a draft until someone accepts it.
- Records every AI call in the audit log with the model and token counts.
- Lets the firm choose the model for each AI feature, or switch AI features off.
What it does not do
- It does not give legal advice, and AI output can be wrong or incomplete.
- It does not replace a lawyer's review or make anyone's supervision duty lighter.
- When AI features are on, the text needed for a request is sent to the AI provider for that feature.
This article is general information about the law, legal practice and technology, current when it was published. It is not legal advice. Laws and guidance change: check the sources linked above and take advice on your own circumstances from a lawyer qualified where you are.
Read next
Client confidentiality and AI tools: a checklist before you paste anything in
The questions to answer about retention, training use, access, hosting and sub-processors before client information goes into any AI tool.
5 minute readShadow AI in law firms: staff are already using it, so what next?
Why bans fail, how to find out what is being used, and how to offer an approved alternative with clear rules.
5 minute readWriting a law firm AI policy: an outline you can adapt
Six sections that a short, usable AI policy needs, with the questions each one should answer.
4 minute read