ChamberOne

How to test an AI tool on your own documents before buying

By the ChamberOne team 5 minute read

Every demonstration uses a document the tool handles well. Your files are different: scanned, annotated, drafted in a hurry and full of local conventions. The only reliable way to know how a tool will perform for your firm is to test it on your own documents, with a plan made before the vendor is in the room. A test like this takes a few days and can prevent an expensive mistake.

Key points

  • Test the specific tasks you want to improve, using files that look like your real work.
  • Protect confidentiality by using closed, anonymised or public documents unless terms cover client data.
  • Set pass marks and a definition of serious failure before the test begins.
  • Plant known errors and unanswerable questions to see how the tool behaves at its limits.
  • Compare tools on equal terms, count the checking time and record your decision.

Decide what you are testing for

Start with the job, not the tool. Which task do you hope it will improve: reviewing contracts, summarising correspondence, extracting dates, searching the precedent bank, producing first drafts? A tool that is excellent at one may be poor at another, and a single overall impression hides the difference.

Write down two or three tasks, and what a good result would look like for each. If you cannot say what good looks like, you will not recognise it in the test. Involve the people who will use the tool, since they know the work.

Choose representative files

Select files that look like your real work, not the neat ones. Include the awkward ones, because they are where tools fail.

  • Documents of the types you handle most often, in ordinary quantities.
  • A few long documents and a few short ones.
  • Scanned or poorly formatted files, and files with handwriting or stamps if they are common in your work.
  • Documents with unusual structure, such as schedules, tables and cross-references.
  • Files that use local terms, local statutes or another language, if relevant.
  • At least one document you know well enough to judge any output instantly.

Protect confidentiality during the test

Testing on real documents raises the confidentiality question at once. Use old, closed matters with names and identifiers removed, or public documents of the same type, unless the vendor has given written terms that cover your client data. Check any client restrictions and your regulator's guidance before uploading anything. Ask the vendor what happens to test files afterwards, and get deletion confirmed in writing.

Keep a note of what you uploaded, where and when.

Define pass marks before you start

Agree your standards in advance, so that a polished demonstration does not move the goalposts. A pass mark should be specific and measurable: for example, that the tool finds every party and date in a set of contracts, or that a summary includes every obligation that you have listed as critical.

Decide also what counts as a serious failure. An invented fact, a misstated amount or an omitted clause that changes the meaning should be treated differently from clumsy wording. Set the rule that a single serious failure on a high-risk task means a fail for that task.

Plant known errors

To see whether a tool catches what it should, give it documents in which you have placed problems yourself. Take a contract and change a date so that it contradicts another clause. Remove a standard protection. Add a definition that is used inconsistently. Introduce a wrong cross-reference. Then see what the tool says.

Include a few traps for a tool that wants to please: ask a question that the documents cannot answer, and see whether it says so or makes something up. A reliable tool will say that the information is not in the document. Record each result.

Errors you plant
  • A date that contradicts another clause
  • A standard protection removed
  • A definition used inconsistently
  • A wrong cross-reference
Traps for a tool that wants to please
  • A question the documents cannot answer
  • A reliable tool says the information is not there
  • An unreliable one makes something up
Planting errors and traps in a tool test

Compare tools fairly

If you are testing more than one product, give each the same documents, the same instructions and the same time. Score the outputs blind where possible, with the reviewer not knowing which tool produced which result. Use a simple scoring sheet and have two people score independently, then compare.

  • Accuracy: were facts, figures and names correct?
  • Completeness: was anything important missing?
  • Traceability: could you see where each point came from in the source?
  • Usefulness: how much editing was needed before you would use it?
  • Time: how long did it take, including checking?
  • Behaviour at the limits: what did it do with something it could not answer?

Count the checking time

A tool that produces output in a minute but needs thirty minutes of verification may not save time. Record how long the checking took for each output, and by whom. Include the time to learn the interface and to correct what it got wrong. This is the figure that decides the business case.

Also test the practical side: how files are uploaded, how results are exported, what happens when a document is too large, and whether support answers a question promptly during the trial.

A hypothetical test day

Suppose a six-lawyer commercial firm tests two tools on ten closed supply agreements with names removed. Two documents are poor scans, and one has a planted contradiction between a payment clause and a schedule. A partner and an associate score each output separately against a sheet. Tool A finds every party and date but misses the contradiction and quietly fills a gap in a termination clause with plausible wording. Tool B misses two dates but flags the contradiction and says that the termination notice period is not stated. The scores do not agree, and that is useful: it shows which failure the firm is more willing to live with.

Decide, and write down why

At the end of the test, gather the scores, the failures and the comments of the people who used it. Decide on the basis of the pass marks that were set at the start. Record the decision and the reasons, including any conditions, such as limiting the tool to certain tasks. If the test shows that a tool is not ready, that is a good result for the cost of a few days. Repeat the test after any major change by the vendor.

Trying ChamberOne on a sample matter

  • Offers a 60-day free trial with no card required.
  • Loads a sample matter so that the screens are not empty.
  • Analyses an uploaded document and lists its type, parties, key dates, clauses and citations, each linked to the source paragraph.

What it does not do

  • The analysis is produced by an AI model and can miss or misread things, so check it against the document.
  • It does not give legal advice or replace a lawyer's review.
  • The document text is sent to the AI provider chosen for the feature.

This article is general information about the law, legal practice and technology, current when it was published. It is not legal advice. Laws and guidance change: check the sources linked above and take advice on your own circumstances from a lawyer qualified where you are.

Read next

How to test a legal AI tool on your own documents