ChamberOne

Red flags in legal AI vendor claims

By the ChamberOne team 5 minute read

A sales deck says the tool has no hallucinations, replaces a team of associates and is completely secure. Each of those sentences should raise your guard. Not because every vendor is dishonest, but because honest vendors know that these claims cannot be proven. Part of the buyer's job is to turn confident slogans into specific statements that can be tested, written down and, if necessary, held against the contract.

Key points

  • Treat zero hallucination claims as a prompt for questions about method, not a fact.
  • A tool that shows its sources and makes checking easy is more trustworthy than one that promises to make checking unnecessary.
  • Replace vague security language with specific questions about location, training, providers, logs and exit.
  • Insist on a trial with your own documents and written answers to the same questions from every vendor.
  • Watch for pressure, lock-in and changes of model without notice.

Why vendor claims need extra scrutiny in law

In most software markets, a bad purchase costs money. In a law firm, it can cost a client their case or the firm its reputation. The people most likely to rely on a claim, busy partners, are also the ones with the least time to test it. Vendors know this, and the competition for attention rewards bold statements.

The remedy is not cynicism. It is a habit of asking what a claim means in practice, what evidence supports it and what happens when it turns out to be wrong.

Red flag: zero hallucination

Language models can produce text that is fluent and false. Techniques such as grounding answers in a set of documents can reduce the problem, and are worth asking about. But a claim that errors have been eliminated is not credible without a clear definition of what was measured and how.

Ask what the vendor means by an error, how it tests for it, on what documents and how recently. Ask whether the tool shows the passage that supports each answer so that you can check it. A tool that makes verification easy is more honest than one that promises to make it unnecessary.

Red flag: replaces lawyers

Marketing that suggests software replaces lawyers either misunderstands legal work or hopes the buyer does. Professional responsibility, advocacy, judgement and client relationships do not move to a tool. What can change is how long some tasks take.

If a vendor claims time savings, ask for the method behind the claim. Which tasks were measured? Who did the work? How was review time counted? A saving that ignores checking is not a real saving. Be equally wary of claims to replace a stage of work without review, which put the risk back on you.

Red flag: vague security

Phrases such as bank-grade security, military-grade encryption and fully compliant say very little. Security is a set of specific controls, and a good vendor can describe them plainly.

  • Where is data stored and processed, and by whom?
  • Is your data used to train any model, and can you opt out in writing?
  • Which AI providers receive your text, under what terms, and in which countries?
  • How is access controlled and logged, and can you see the logs?
  • What happens to your data when you leave, and how long does it stay in backups?
  • Which independent assurance reports exist, and can you read the actual report rather than a badge?

Red flag: results with no method

Be careful with figures about accuracy, speed or savings that come without detail. A statistic that cannot be traced to a method, a data set and a date is a marketing device. The same applies to testimonials from unnamed firms and logos on a slide with no evidence of actual use.

The best evidence is your own. Ask for a trial on your documents, and judge it by the standards you would apply to a junior colleague's work. If the vendor resists a fair test, ask why.

Red flag: pressure and lock-in

Be wary of urgency that has no reason behind it, such as a discount that expires on Friday or a pilot that automatically becomes an annual contract. Ask what happens at the end of a trial, whether pricing is fixed or metered, and what you can take with you. A vendor confident in the product does not need to trap you.

Check also for terms that allow the vendor to change models or providers without notice. Models change, and outputs change with them. Ask how you will be told and whether you can pin a version for important work.

Questions to put to every vendor, in writing

Send the same questions to each vendor and compare the written answers. Vague or evasive replies are information. Keep the answers with your procurement notes, since they may be useful later.

  • What exactly does the tool do, and what does it not do?
  • Can it show the source for each answer?
  • What is sent to which AI provider, and what are the terms with that provider?
  • Can I turn AI features off, and does anything leave my system when they are off?
  • How do I export my data, in what format and at what cost?
  • What happens when the tool is wrong, and what does your contract say about it?

Reading the contract as well as the website

Marketing pages and contracts often say different things. A website may promise that data is never used for training while the terms reserve a broad licence to use customer content. Read the order form, the terms of service, the data processing terms and any list of sub-processors. Where the contract is weaker than the slogan, ask for the slogan to be written into the contract.

Ask who is liable if the tool produces something that causes loss, and whether liability is capped at a figure that bears any relation to the risk. Many vendors will say, quite reasonably, that the lawyer remains responsible for review. That is fair, but it should be a conscious choice by the firm, made after reading the clause, and not a surprise discovered later.

Putting the answers to work

A good vendor will welcome specific questions, and will say so when the answer is that a feature does not exist. Pay attention to how limits are described. Plain statements such as 'the output is a draft and a lawyer must review it' are a good sign. Treat a vendor that cannot name any limits as one that has not thought about them.

Red flags
  • Claims that errors have been eliminated
  • Software said to replace lawyers
  • Security phrases with no detail
  • Figures with no method, data set or date
  • Pressure and lock-in
Good signs
  • Shows the passage behind each answer
  • Says plainly the output is a draft a lawyer must review
  • Describes security controls in plain terms
  • Names its limits and says when a feature does not exist
  • Allows a fair trial on your documents
Reading a legal AI vendor's claims

This article is general information about the law, legal practice and technology, current when it was published. It is not legal advice. Laws and guidance change: check the sources linked above and take advice on your own circumstances from a lawyer qualified where you are.

Read next

Red flags in legal AI vendor claims | ChamberOne