ChamberOne

Recording AI queries: building a research audit trail

By the ChamberOne team 4 minute read

Months after a memo goes out, a client asks how you reached a view. Or a court asks where a citation came from. Or a partner reads an old file and wonders whether anyone checked the point. If AI helped with the research, the honest answer depends on what you kept at the time. Most firms keep the final memo and nothing else. A research audit trail is the small set of records that lets you reconstruct how an answer was produced: the question asked, what came back, which sources were opened and who checked what. It costs a few minutes per task and is most valuable on the day you need it.

Key points

  • Record the prompt, tool, output relied upon, sources opened, checks done and the decision taken.
  • Keep the record on the matter, and decide in writing whether it forms part of the file.
  • Use a short, standard verification note on every piece of work where AI helped research.
  • Keep evidence of errors caught; it shows the checks work and informs tool choice.
  • Review a sample each quarter and avoid over-collecting exploratory material.

What an audit trail is for

The purpose is not bureaucracy. It serves three practical needs. First, accountability: you can show that a person reviewed the output and that the review was real. Second, error recovery: if a source turns out to be wrong, you can find every piece of work that relied on it. Third, learning: you can see which kinds of questions the tools handle well and which they get wrong.

A trail also protects the individual lawyer. A junior who can show exactly what they asked, what they were given and what they verified is in a far better position than one who says 'I think I checked it'.

What to keep

Keep enough to retrace the steps, and no more than that. Every extra item is something to store, protect and eventually delete.

  • The question or prompt, in the words actually used, with the date and the person who asked.
  • The tool and model used, where the system records it.
  • The output, or the part of it that was relied upon.
  • The sources the output pointed to, and whether each was opened and read.
  • The checks done: who verified which authority, in which database or official source.
  • The decision: what was accepted, what was changed and what was rejected.

Where to keep it

The best place is on the matter, next to the work it supports. A separate folder on someone's laptop will not be found when it is needed. If your system records AI calls automatically, such as who made the request and when, use that as the backbone and add a short human note on checking.

Decide whether the record is part of the client file or an internal working record. Rules on what belongs to the file differ between jurisdictions, so check the guidance from your regulator or bar and make a written decision. Whatever you choose, apply your retention policy to it.

A verification note that takes two minutes

The most useful record is a short, standard note attached to any piece of work where AI contributed to research. It can be five lines. Suppose a three-lawyer firm uses this template for every research memo: tool used and task; sources the tool named; which of those were found and read in an official or trusted database; points the tool made that could not be verified and were removed; reviewer's name and date.

Consistency matters more than detail. A short note on every file beats a long note on a few.

Why prompts matter

Two people can ask about the same legal point and get different answers because their questions differ. A leading prompt, such as 'explain why the clause is unenforceable', invites an answer that assumes the conclusion. Keeping the prompt shows whether the question was neutral, and it lets a reviewer rerun it.

Prompts also reveal what information the person put into the tool. If a prompt contains client names or confidential facts, you will want to know that, both for confidentiality and for conflicts.

Handling outputs that were wrong

Do not delete the evidence of an error. If a tool gave a wrong citation and the reviewer caught it, a one-line note saying so is useful. It shows the check worked, and it contributes to your picture of the tool's reliability.

Where a wrong output reached a client or court, record what happened, who was told and what was done to correct it. Take advice from your risk partner or insurer on obligations to notify; do not assume.

Do
  • Keep a one-line note of the error that was caught
  • Record what happened, who was told and what was done if it reached a client or court
  • Take advice from your risk partner or insurer on notification
Don't
  • Delete the evidence of an error
  • Assume what your obligations to notify are
When an AI output turns out to be wrong

Using the trail

A trail that nobody reads is wasted effort. Build in a periodic review. Once a quarter, a partner or compliance lead pulls a small sample of matters where AI helped and reads the notes. Are checks being done? Do the notes match the memo? Do certain people or tasks produce more rejections?

The sample review also tests the trail itself. If you cannot reconstruct an answer from the notes, the notes are too thin, and it is better to find that out now than when a client asks.

  1. 1Pull a small sample of matters where AI helped
  2. 2Read the notes and ask whether checks were done
  3. 3Compare the notes with the memo
  4. 4Look for people or tasks with more rejections
  5. 5Thicken the notes if an answer cannot be reconstructed
  6. Then back to step 1.
The quarterly review of the audit trail

Suppose a challenge arrives

Imagine a client questions a conclusion in a memo written six months ago. With a trail, the partner opens the matter, sees the question as asked, the tool and the sources it named, and a note showing which authorities were read and by whom. In ten minutes the partner can say what was done and what was not, and where the conclusion came from. Without it, the firm relies on memory and goodwill.

Limits

A record shows what was done, not that it was done well. A box ticked beside 'verified' proves nothing unless someone actually opened the case. Audit trails also cannot capture reasoning that happened in a lawyer's head, so encourage short comments on why a source was relied on. And logs of AI calls record that a request occurred; they do not tell you whether the answer was right.

Be careful not to over-collect. A trail that stores every exploratory prompt forever creates a pile of confidential material that must be protected and eventually deleted. Set a rule: keep what supported a conclusion, and clear exploratory sessions on a schedule your policy allows.

Audit records in ChamberOne

  • Every read and write is recorded automatically with who, when, IP address and before and after values.
  • Every AI call is recorded in the audit log with the model and token counts.
  • Audit events can be filtered by person and action, exported and are kept for 7 years by default.

What it does not do

  • The log records that an AI call happened; it does not show whether the answer was right. A lawyer still checks the output.
  • It does not write your verification notes for you.
  • Tamper-evident hashing is on the roadmap and is not in the product today.

This article is general information about the law, legal practice and technology, current when it was published. It is not legal advice. Laws and guidance change: check the sources linked above and take advice on your own circumstances from a lawyer qualified where you are.

Read next

AI research audit trail for law firms | ChamberOne