ChamberOne

Can AI predict case outcomes? Limits and sensible uses

By the ChamberOne team 5 minute read

Clients ask the question in the first meeting: what are my chances? Software vendors say they can answer it with a number. A litigator who has run enough cases knows why the number should make you nervous. The honest position is somewhere in the middle. Pattern analysis of past decisions can inform a conversation about risk. It cannot stand in for the judgement that turns a pattern into advice on one client's claim.

Key points

  • Prediction tools describe past cases; they do not know your facts or your judge.
  • Published decisions are a biased sample, because settled and short cases leave little trace.
  • Use patterns to triage and to challenge your assumptions, not to give a single number to a client.
  • Build settlement ranges from your own analysis first, then compare.

What a prediction tool is really doing

Strip away the interface and these tools do one of two things. Some count: they look at past cases with similar features, such as court, judge, claim type and parties, and report how those cases ended. Others use a model trained on case text to estimate a likelihood from the features it finds.

Both approaches describe a past population. Whether your case belongs to that population is the real question, and the tool cannot answer it.

The data problem

Start with what is missing. In many of the jurisdictions our readers practise in, only some judgments are published, and published judgments are not a random sample. Settled cases leave no decision. Cases that were struck out early, withdrawn or decided from the bench may never appear. Courts differ in how much they put online and how well it is indexed.

The result is selection bias. A database of reported decisions over-represents cases that were contested to the end, that raised a point of law, or that someone thought worth reporting. If your matter is a routine dispute that would normally settle, the tool is comparing it with the unusual ones that did not.

Why similar cases are not the same case

Outcomes turn on things that rarely reach a database as clean fields:

  • The quality and consistency of witnesses, which only emerges in the room.
  • Documents that are unhelpful or missing.
  • How opposing counsel behaves and what the other side can afford.
  • Law that has changed since the older decisions.
  • The particular judge's approach on this day to this evidence.

Pattern spotting is not advice

There is a useful distinction between information and advice. A tool can tell you that, among the cases it holds, claims of this type most often fail on limitation. That is information. Advising a client means asking whether limitation is a live issue on these facts, what the documents show, and what the client is prepared to risk. That is judgement, and it carries professional responsibility.

Be wary of a single percentage or confidence score presented to a client. Even if the arithmetic is sound, it creates a false sense of precision. A lawyer who passes it on takes responsibility for it. Check the guidance from your regulator or bar on what you must tell clients about the basis for your advice.

Information (what a tool offers)
  • Reports patterns in the cases it holds
  • For example, that claims of this type most often fail on limitation
Advice (what a lawyer gives)
  • Asks whether limitation is a live issue on these facts
  • Checks what the documents show
  • Weighs what the client is prepared to risk
  • Carries professional responsibility
Information and advice are different jobs

Sensible uses

There are places where this kind of analysis earns its keep, as long as you treat it as one input among several.

  • Early triage. Does this type of claim tend to succeed, fail or settle, and what issues decide it? The answer shapes where to look first, not what to advise.
  • Spotting arguments. Patterns in how courts have dealt with an issue can suggest arguments you had not considered.
  • Checking your own instinct. If a tool's pattern conflicts with your view, ask why. Sometimes you are right and the data is thin; sometimes you have been optimistic.
  • Preparing a risk conversation. Presenting a range of possible outcomes with the reasons behind each is more honest than one figure.

Using it carefully for settlement ranges

Settlement is where the temptation is strongest. A way to use data without over-relying on it is to build your range from your own analysis and then compare it with whatever the tool suggests.

  1. 1

    Write down best, likely and worst outcomes for each head of claim, with the reasons and the evidence for each.

  2. 2

    Estimate costs to trial and the chance of recovering them, using your own matter history where you have it.

  3. 3

    Use any prediction tool only to challenge a step, such as how courts have treated a particular type of loss.

  4. 4

    Where the tool and your analysis differ, find out why before you move either.

  5. 5

    Present the client with a range, the assumptions behind it and what would change it, and record that you did.

Questions for a vendor

  • What cases are in the data, from which courts and which years?
  • How are unreported and settled cases treated?
  • Can I see the cases behind a result and read them?
  • How has the tool been tested, and by whom?
  • What does the tool do when my case is unlike anything in its data?

If the answers are vague, treat the output as marketing and keep your own judgement in charge.

Talking to clients about uncertainty

Clients often want certainty because they are anxious, not because they are naive. A calm explanation of why certainty is not available usually serves them better than a number that later proves wrong.

Explain that outcomes depend on evidence not yet tested, and that your assessment will move as the case develops. Say which events would change it: disclosure of a key document, a witness statement that does not hold up, an adverse ruling on a preliminary point. Offer to revisit the assessment at those points.

If a client brings a printout from a prediction website, do not dismiss it. Ask what data it used, then walk through where your facts differ. That conversation builds trust, and it protects you if the case goes the other way.

When the tool disagrees with you

Sooner or later a tool will point one way and your experience the other. Write down your reasons before you look at the tool's output, so that you do not talk yourself into its view. Then compare. If the tool has found a pattern you missed, test it against the facts. If it has not, make a note of why, since that note may help the next lawyer who sees the same pattern.

This article is general information about the law, legal practice and technology, current when it was published. It is not legal advice. Laws and guidance change: check the sources linked above and take advice on your own circumstances from a lawyer qualified where you are.

Read next

Can AI predict case outcomes? Limits and uses | ChamberOne