How to Avoid AI Hallucinations in Legal Work

6 min readNovember 2, 2026

Gist

An AI hallucination is output that sounds right but is false: a case that doesn't exist, a quotation nobody wrote, a holding that says the opposite of what the court decided. You reduce the risk of hallucinations by grounding the AI in real documents, requiring a source for every claim, and having a lawyer check every citation against the primary source before anything is filed or sent. This guide includes a worked contract example, explains why hallucinations happen and what courts are doing about them, and gives eight steps for checking AI output.

Why AI tools hallucinate

Large language models write by predicting the most likely next words. They don't look anything up unless they are connected to a source, so when a model doesn't know an answer it can still produce one that reads well. Legal citations make this worse: they follow a predictable format, so a model can generate a case name, reporter and page number that look exactly like the real thing.

Connecting the model to real documents (often called retrieval or grounding) reduces hallucinations, but it doesn't remove them. A Stanford RegLab and HAI study tested leading AI legal research tools whose vendors had marketed them as avoiding hallucinations. The tools still hallucinated on between 17% and 33% of the queries tested.

What happens when hallucinations reach a court

The best-known case is Mata v. Avianca (S.D.N.Y. 2023). Lawyers filed a brief citing six cases that ChatGPT had invented, and then defended the citations when the court questioned them. In June 2023 the court imposed a $5,000 sanction on the lawyers and their firm, and ordered that the sanctions decision be sent to the judges whose names had been attached to the fake opinions.

Mata was the start, not an outlier. Damien Charlotin's AI Hallucination Cases database had identified more than 2,200 court and tribunal decisions involving hallucinated content by November 2026, many of them involving lawyers rather than self-represented parties. Sanctions now range from fines and fee awards to referrals to disciplinary authorities and removal from a case.

The ethics rules put the responsibility on the lawyer, not the tool. ABA Formal Opinion 512 says lawyers must review AI output for accuracy, and filing false citations engages the duties of candor to the tribunal and meritorious claims. "The AI wrote it" is not a defense.

Worked example: an answer that goes beyond the contract

This fictional example shows a common failure: an AI answer drops a notice requirement and a payment obligation while sounding certain. The document and answers below are illustrative, not a customer result or advice about a real agreement.

1. The question and unsupported answer

Question: Can Acme terminate this services agreement for convenience immediately, without paying anything further?

Unsupported AI answer: “Yes. Acme can terminate for convenience at any time, effective immediately, with no further payment obligation.”

That answer gives a confident conclusion without identifying a provision. It also adds two claims the source below does not support: immediate termination and no further payment.

2. The underlying document

Acme–Northstar Services Agreement (fictional), section 8.2 — Termination for convenience:

“Customer may terminate this Agreement for convenience upon thirty (30) days' prior written notice to Provider. Customer remains liable for all fees due for Services provided before the effective date of termination.”

This is an excerpt, not the complete agreement. Other provisions or amendments could affect the answer, so the reviewer must check the full document before advising the client.

3. An answer cited to the source

Cited answer: “Section 8.2 permits termination for convenience on thirty days' prior written notice and keeps the customer liable for fees due for services provided before termination takes effect. This excerpt does not support immediate termination with no further payment. Check the full agreement and any amendments for other applicable provisions.” Source: Acme–Northstar Services Agreement, section 8.2.

The citation helps the lawyer reach the text. It does not establish that the answer is correct: the cited passage still has to support each material claim, and an excerpt may leave out an exception elsewhere in the agreement.

4. The lawyer's verification steps

  1. Confirm the document and parties. Check the current executed agreement and amendments, and confirm that Acme is the defined Customer.
  2. Read section 8.2 directly. Match the cited wording to the source and confirm the thirty-day notice requirement and continuing fee obligation.
  3. Check the rest of the agreement. Review definitions, notice procedures, other termination rights, payment terms, schedules and amendments for provisions that change the analysis.
  4. Check the matter facts. Establish whether and how notice was given, the relevant dates, services already provided and any outstanding fees. The excerpt alone cannot determine an effective termination date or an amount due.
  5. Review the proposed advice. Correct unsupported statements, identify unresolved questions and have the responsible lawyer approve the final communication before it is sent.

The same method applies to cited cases and statutes: open the primary source, confirm it says what the answer claims, and check its context and current legal status. A real citation can still be used to support a proposition the source never makes.

8 ways to avoid AI hallucinations

1. Ground the AI in real documents

For legal substance, use tools that answer from a defined set of sources, such as your files, the record or a research database, rather than an open-ended chatbot answering from memory.

2. Require a source for every claim

Prefer tools that link each answer to the document and passage it came from. An answer you can't trace is an answer you have to research from scratch anyway.

3. Check every citation in the primary source

Pull every case, statute and regulation. Confirm it exists, that it says what the draft claims, and that it is still good law. A citator check alone doesn't confirm that the case supports your proposition.

4. Verify quotations word for word

Models paraphrase and then present the paraphrase in quotation marks. Match every quotation against the source text, including pin cites.

5. Ask narrower questions

AI is most reliable when it extracts, compares and summarizes material you give it ("list every change-of-control clause in these agreements") and least reliable when asked open questions about the law in general. Break big questions into checkable steps.

6. Treat fluent output as unverified

Hallucinations don't look different from correct answers. Confident, well-written output deserves the same scrutiny as a first draft from a new associate. When in doubt, ask the tool to show the exact passage it relied on.

7. Build review into the workflow

When AI or an AI agent handles a multi-step process, put a lawyer's review before anything leaves the firm: before filing, before sending to a client, before a redline goes to the other side. Decide in advance who reviews what.

8. Put it in your AI policy and train everyone

Verification rules only work if everyone knows them, including paralegals and staff. Our free AI policy template for law firms includes a verification section you can adopt as is.

A pre-filing checklist

How Gist approaches accuracy

Gist answers from your actual documents, and every answer links back to the document and passage it came from, so you can verify before you rely. Gist's agents work from your own precedents and standards rather than generic templates. No tool removes the lawyer's duty to check the work, but tools that show their sources make that check faster. Book a demo to see it on your own documents.

More from Gist

Put AI to work the way
your team works.

Start free on the Gist platform. When there's work you have not been able to
automate with AI, our deployment team builds it with you.

Book a demo Start Free