ChamberOne

What AI document analysis extracts well, and what it misses

By the ChamberOne team 5 minute read

Analysis tools are good at the parts of a document that are written down plainly and in the usual places. They are poor at what a document implies, what is handwritten in the margin and what sits in a table that the scanner turned sideways. Knowing this split lets a paralegal or associate use the tool for what it does well and look for the gaps where it does not. This article sets out the split and a routine for finding the gaps.

Key points

  • Analysis tools do best on explicit, labelled, conventional content: parties, stated dates, headed clauses and citations.
  • They are weakest on implied terms, calculated dates, handwriting, tables and poor scans.
  • An extraction lists what the tool recognised, not everything in the document.
  • Check the original beside the output, and record the page or paragraph for every fact you use.
  • For large sets, sample and look for clusters of errors in particular document types.

What tends to come out well

Analysis tools generally do best on information that is explicit, labelled and conventional in form.

  • The type of document: lease, loan agreement, witness statement, letter.
  • The parties and their roles, where they are named in the usual place.
  • Stated dates, such as the date of the agreement, the term and notice periods.
  • Headed clauses, such as termination, confidentiality and governing law.
  • Citations to statutes and cases that appear in the text.

What tends to go wrong

The failures are quieter than a missing field. They are plausible, specific and wrong.

  • Implied terms and obligations that arise from the structure of the document or from the law rather than from a clause.
  • Dates that depend on a calculation, such as thirty days after notice or the end of the following quarter.
  • Parties who are described but not named, or whose name changes between the front page and the schedule.
  • Handwriting, stamps, signatures and annotations, which are often skipped or misread.
  • Tables, forms and multi-column layouts, where text can be read in the wrong order.
  • Poor scans, where character errors turn one figure or name into another.
Tends to come out well
  • The type of document.
  • Parties named in the usual place.
  • Stated dates, such as term and notice periods.
  • Headed clauses and citations in the text.
Tends to go wrong
  • Implied terms and obligations.
  • Dates that depend on a calculation.
  • Handwriting, stamps and annotations.
  • Tables and multi-column layouts.
  • Poor scans.
Where document analysis tends to succeed and fail

Why the gaps are hard to see

A tool that returns a tidy list of parties, dates and clauses does not tell you what it left out. There is no empty field to alert you, and a list of five key dates looks complete even when the document contains eight. The risk lies in treating the output as an inventory of the document, when it is only an inventory of what the tool recognised.

Suppose a paralegal analyses a commercial lease that has a rent review clause in a schedule at the back, set out as a table. The tool lists the term, the parties and the break date, all correct, and says nothing of the review. The lease's most expensive clause is simply absent from the summary. Nothing signals the omission, because the output looks tidy. Only a person who knows that leases usually contain a review clause, and looks for it, finds the gap.

A routine for spotting the gaps

Build a short check into every document set before anyone relies on the extraction.

  1. 1

    Open the original beside the extraction and confirm that the first and last pages are present and readable.

  2. 2

    Check the parties against the signature block and the definitions, not just the front page.

  3. 3

    Scan the document for numbers and dates yourself and tick off each one in the extraction. Anything missing is a prompt to read that passage.

  4. 4

    Look at every table, schedule and annex. Confirm the content matches what the tool recorded.

  5. 5

    Note every handwritten, stamped or illegible area and read it by eye.

  6. 6

    Ask what the document would say if a clause were missing, and check for it.

Link every extracted item back to its source

The most useful feature in any analysis tool is the ability to jump from an extracted item to the paragraph it came from. If you can click from a date to the sentence, checking takes seconds. If the tool gives no such link, the check takes as long as reading the document, and you should weigh whether the tool is saving anything.

When you record an extracted fact in a note or a chronology, record the page or paragraph with it. That habit makes later review quick for everyone.

Where the tool offers a risk flag on a clause, read it as a prompt to look at that clause with care. A flag is not a conclusion, and the absence of a flag is not clearance. A clause the tool considers routine may be unusual for your client, and a clause it flags may be perfectly standard in the sector. Context lives in your head and in the instructions, not in the document.

Sets of documents add their own risks

When many documents are analysed at once, small error rates add up. A tool that is right most of the time on each document will still leave errors somewhere in a set of hundreds. Sample the output. Pick a handful of documents at random, check them fully, and record what you found. If errors cluster in a particular type of document, such as scanned letters, treat that type with extra care.

Look also at consistency across the set. The same party recorded under three spellings, or the same contract given two different dates, is a signal worth investigating.

Keep a log of the errors you find, even small ones, with the type of document and the type of error. A few weeks of notes will tell you where the tool can be trusted with a light check and where it needs a full read. It will also give you something concrete to raise with the supplier.

Handling confidential documents

Document analysis usually means sending the text to an AI provider. Before you upload client documents, find out what the provider does with them, where they are processed and how long they are kept. Check the guidance from your regulator or bar on confidentiality, and apply the firm's policy on approved tools.

Decide, before the first upload, which categories of document may be analysed this way and which may not. Some clients restrict what may leave your systems, and some documents, such as those subject to a confidentiality undertaking, need extra thought. Record the decision so nobody has to guess at midnight.

How ChamberOne handles this

  • You can upload a PDF, Word, Excel, CSV, HTML, text or markdown file to a matter.
  • The analysis lists the document type, parties, key dates, clauses and citations, each linked to the paragraph it came from.
  • Risk flags on clauses are suggestions a partner can override.
  • Citations can be found again from other matters.

What it does not do

  • The analysis is produced by an AI model and can miss or misread things. Check it against the document.
  • It does not give legal advice, and it does not replace a lawyer's reading of the document.
  • The document text is sent to the AI provider chosen for this feature. The Security and AI page says what is sent and on what terms.

This article is general information about the law, legal practice and technology, current when it was published. It is not legal advice. Laws and guidance change: check the sources linked above and take advice on your own circumstances from a lawyer qualified where you are.

Read next

AI document analysis for lawyers: what it extracts, misses