ChamberOne

E-discovery and technology-assisted review explained for non-specialists

By the ChamberOne team 5 minute read

A litigator who has run mostly paper cases is told that the other side has three million emails and that the review will be done with technology-assisted review. The phrase sounds like a black box. It is not, and a lawyer does not need to understand the mathematics to supervise it well. What matters is knowing what the software is doing, how its results are tested, and what record you will need if the process is questioned. This article explains those points in plain terms.

Key points

  • Technology-assisted review ranks or classifies documents so people can focus on those likely to matter.
  • Predictive coding learns from labelled examples. Generative review follows a written instruction and gives reasons.
  • Validation means sampling what was set aside and checking what was missed, not trusting the tool.
  • Keep a short protocol recording scope, decisions, samples, results and handling of privilege.
  • Check the disclosure rules of the court concerned and tell the other side early where the rules allow.

The basic problem

In many disputes, each side must identify, review and disclose documents that are relevant. When the pool is large, reading every item is slow, costly and often disproportionate. Keyword searches reduce the pile, but they are crude: they miss documents that use different words and they sweep in many that are irrelevant.

Technology-assisted review, often shortened to TAR, uses software to help decide which documents are likely to be relevant, so human reviewers can concentrate on those and on a sample of the rest.

Suppose a litigation team in a regional firm is told its client holds a mailbox archive far too large to read. Nobody on the team has run a technology-assisted review before. The team does not need to become expert in statistics. It needs one person who can explain, in plain words, what the method will do, what it will not do and how the result will be tested.

Predictive coding: learning from examples

The longer-established approach is sometimes called predictive coding. Senior reviewers read a sample and mark each document as relevant or not. The software learns from those decisions and scores the remaining documents by how likely they are to be relevant. Reviewers check more samples, correct the software where it is wrong, and the scoring improves.

The result is a ranking. Documents at the top are reviewed by people, and a cut-off is chosen below which documents are treated as not relevant, subject to testing.

The quality of the ranking depends heavily on the people who train it. If the first reviewers are junior or the instructions about relevance are vague, the software learns the confusion. Put your most experienced reviewers on the training stage and write down the criteria they apply.

Generative review: following instructions

A newer approach uses large language models. Instead of learning from many labelled examples, the model is given a written description of what is relevant and asked to assess each document against it, often with a short explanation. This can start faster and gives a reason a human can read.

The trade-offs are different. The reasons can be persuasive and wrong. Results may vary with how the instruction is worded. Processing text at volume has cost and data-handling implications. Those who use it need to test it as carefully as any other method.

A practical point: because the explanation is written in fluent prose, it is easy to read it, agree and move on. Reviewers should sometimes mark a document without reading the model's reason first, and then compare. That keeps the human decision independent.

Predictive coding
  • Senior reviewers mark a sample as relevant or not.
  • The software learns and ranks the remaining documents.
  • A cut-off is chosen, subject to testing.
  • Quality depends on the people who train it.
Generative review
  • The model gets a written description of what is relevant.
  • It assesses each document, often with a short reason.
  • The reasons can be persuasive and wrong.
  • Results may vary with the wording of the instruction.
Predictive coding against generative review

What courts tend to care about

Rules and practice directions on disclosure differ between countries and between courts, and some deal expressly with technology. Check the rules and any practice direction in the court where your case sits, and take specialist advice where the volumes justify it.

In general terms, the questions a court or the other side is likely to raise are consistent. Was the approach reasonable and proportionate? Was it explained? Was it tested? Can the party show what it did?

Validation: showing that it worked

Validation means testing the result rather than assuming it. The usual idea is to take a random sample from documents that the process treated as not relevant, have people review it, and see how many relevant documents were missed. If too many were, the process is adjusted and tested again.

A lawyer should be able to answer three questions in plain words: what did we test, how big was the sample and who reviewed it, and what did we decide as a result? Ask a specialist or your provider to explain the statistics in terms you can repeat.

Documenting your process

Keep a running protocol that you can produce if challenged. It does not need to be long.

  • The scope of the review and the issues for relevance, as they were written at the time.
  • Who made the key decisions, such as the choice of method and cut-off.
  • How the method was trained or instructed, and any changes during the review.
  • What samples were taken, who reviewed them and what they found.
  • How privileged and confidential documents were identified and handled.
  • Any agreement with the other side about method.

Raising it with the other side

Where the rules allow, telling the other side early that you intend to use technology, and explaining the outline of your method, often avoids disputes later. Parties sometimes agree a protocol covering sampling and exchange of statistics. An open approach is generally easier to defend than a surprise.

Avoid claims about accuracy you cannot support. Do not tell a court or an opponent that technology makes the review complete or error-free. It reduces the number of documents people must read. It does not remove the need to test.

Keep the numbers you give honest and in context. A figure for how many relevant documents were found means little without saying how it was measured and on what sample. If you do not understand a statistic, ask your provider to explain it before you repeat it to a court.

Privilege and confidentiality

Sending disclosure documents to any external system raises questions about confidentiality, privilege and data protection. Find out where the data goes, who can access it and whether it is retained. Treat privilege review as a human decision, with software at most helping to find candidates. Check your regulator's guidance and any client restrictions before uploading material.

This article is general information about the law, legal practice and technology, current when it was published. It is not legal advice. Laws and guidance change: check the sources linked above and take advice on your own circumstances from a lawyer qualified where you are.

Read next