ChamberOne

Choosing AI models and firm-level controls: what a practice should decide

By the ChamberOne team 4 minute read

Most firms meet AI as a feature that arrives switched on. Someone clicks it, it works, and it spreads. Three decisions are easier to make before that happens than after. Which model should do which task? When should AI be off altogether? And how will you see what was used, by whom and for what? This article is for managing partners and IT leads, and it treats the choice of model as a business decision, not a technical one.

Key points

  • List tasks first, rate the cost of a mistake for each, and let that set how much checking is needed.
  • Test candidate models on your own kind of documents and have a lawyer mark the errors.
  • Decide in writing when AI is off, and check how your tools can actually enforce it.
  • Keep a usage record of who, what, when and which model, and review it on a schedule.

Start from tasks, not from model names

Model names change every few months and comparisons go out of date quickly. Tasks are stable. List the jobs you want help with: summarising a long file, pulling dates and parties from a contract, answering a question about a matter, drafting a first version of a letter.

For each task, note how bad a mistake would be and who would catch it. A summary that a lawyer will compare with the source can tolerate more error than a draft that goes to a client with only a quick read. That risk rating, not the brand of the model, should decide how much checking each task needs.

How bad a mistake would be →
Most checking

A serious mistake that nobody is likely to catch needs the heaviest checking

Check with care

A serious mistake that a reviewer will probably catch still needs a careful read

Light checking

A minor mistake that is unlikely to be caught needs little checking

Can tolerate error

A summary a lawyer compares with the source can tolerate more error

Chance that someone catches a mistake →
Deciding how much checking a task needs

What to compare when you pick a model

You do not need a benchmark lab. You need a small test using your own kind of work.

  • Take three or four real documents from closed matters, with identifying details removed or with the client's permission.
  • Run the same task on each candidate model and keep the outputs side by side.
  • Have a lawyer who knows the documents mark errors, omissions and anything invented.
  • Note how each model behaves when it does not know. A model that says so is safer than one that fills the gap.
  • Check speed and cost only after quality, because a quick wrong answer costs more than a slow right one.

Why per-feature choice is useful

One model rarely suits every job. A firm may find that one model reads long contracts more carefully while another writes cleaner short emails. If your software lets you choose the model for each AI feature, you can match the model to the task and change it later without retraining the team.

It also makes it easier to respond to news. If a provider changes its terms or has a problem, you can move one feature rather than the whole firm. Ask any vendor whether the choice is yours or theirs, and how you would be told if they changed it.

Deciding when AI should be off

Some matters do not suit AI at all. A client may have asked you not to use it. A matter may involve material under a statutory secrecy duty, or a person whose identity must not leak. A wall around a sensitive transaction may mean even the thought of text leaving the firm is unacceptable.

Write down the rules in advance. For instance: AI is not used on matters flagged as sensitive, and the matter partner decides the flag. Then check how that rule is applied in your tools. Some tools offer a firm-wide off switch and not a per-matter one. If that is your situation, the policy needs another way to enforce it, such as restricting who can see sensitive matters.

Record usage so you can review it

A policy you cannot check is a hope. Ask what record your tools keep. At minimum, you want to know who used an AI feature, on which matter, when, and which model handled it. Cost or volume counts are useful for budgeting. Without any of this, you cannot answer a client who asks whether AI touched their file.

Decide who looks at the record and how often. A monthly review by a compliance lead is usually enough to spot unusual patterns: one person using it heavily, use on a restricted matter, a feature nobody uses and you are paying for.

Who decides, and who can change it

Give model choice and the off switch to a small group of named people, usually the managing partner and the IT or compliance lead. Everyone else should be able to see what is on, but not change it. Record each change with a date and a reason, so that a year later you can explain why a feature was switched off or a model changed.

Questions to put to any vendor

  • Which providers and models sit behind each feature, and will you tell us before that changes?
  • Is our text stored by the provider, for how long, and is it used to train anything?
  • Where is it processed, and does it leave the country our data is held in?
  • Can we switch features off for the whole firm, and can we limit them for particular matters?
  • What do we see afterwards: a log of who used which feature, when and with which model?
  • How do we export that log if a client or regulator asks?

Where this does not help

Choosing the best model does not remove the need to review output. A better model makes fewer mistakes, and the mistakes it does make can be harder to spot. Treat every output as a draft that needs a lawyer. Check, too, that the people using the tool understand why a model was chosen. A team that knows a feature was picked for reading long contracts will not use it to write client advice from scratch. Model choice also does not answer questions about data handling. Those need the vendor's terms, and they matter more than any benchmark.

How ChamberOne handles these decisions

  • Choose the model for each AI feature, GPT-5 or Claude, or switch AI features off for the firm.
  • Each AI output stays a draft until someone accepts it.
  • Every AI call is recorded in the audit log with the model and token counts.

What it does not do

  • AI output can be wrong or incomplete. It does not replace a lawyer's review.
  • With AI features on, the text needed for a request is sent to the provider for that feature, OpenAI or Anthropic, outside the country where ChamberOne is hosted.
  • The switch described is for the firm's AI features. To limit who sees a sensitive matter, use a Restricted matter.

This article is general information about the law, legal practice and technology, current when it was published. It is not legal advice. Laws and guidance change: check the sources linked above and take advice on your own circumstances from a lawyer qualified where you are.

Read next

Choosing an AI model and firm controls for a law firm