We evaluated Gist configurations using GPT-6.1 Sol and Opus 5.5 on two main legal tests: contract processing and legal research. Both configurations completed 100% of planned web research tasks. In the latest graded contract processing results, GPT-6.1 Sol passed 84.9% of graded criteria and Opus 5.5 passed 89.5%. Mean coverage of essential legal research points was 75.7% and 70.0%, respectively.
Contract Processing
The document dataset includes drafting (20.0%), analysis (55.0%), review (20.0%) and NDA negotiation (5.0%). The table groups results by those categories.
- Drafting20%
- Analysis55%
- Review20%
- NDA negotiation5%
- Drafting: Creating and editing corporate governance documents, employment agreements, settlement agreements and limited partnership agreements from supplied materials.
- Analysis: Extracting terms, obligations, deadlines and client facts from employment, property, financing, intellectual property and family-law documents, and summarizing risks and discrepancies.
- Review: Cross-checking purchase agreements, financing commitments, closing checklists, engagement letters, matter plans and securitization term sheets for inconsistencies, missing requirements and risks.
- NDA negotiation: Developing compromise positions, drafting confidentiality clauses and explaining concessions using supplied playbooks and deal documents.
-
GPT-6.1 Sol84.9%
-
Opus 5.589.5%
| Category | GPT-6.1 Sol | Opus 5.5 |
|---|---|---|
| Drafting | 80.5% | 85.8% |
| Analysis | 88.2% | 91.8% |
| Review | 77.3% | 85.1% |
| NDA negotiation | 88.9% | 92.6% |
Drafting includes:
- Creating new documents: Corporate governance documents and employment agreements. GPT-6.1 Sol: 85.5%; Opus 5.5: 93.2%.
- Editing existing documents: Settlement agreements and limited partnership agreements. GPT-6.1 Sol: 77.0%; Opus 5.5: 80.6%.
Each pass rate is the number of passed criteria divided by graded criteria within that category. Criteria have equal weight, so tasks with more criteria contribute more to the category score. Ungraded criteria are excluded. Missing deliverables can fail criteria; they are not silently excluded.
Legal research
The legal research dataset includes questions that require web search (49.3%) and questions that do not (50.7%). The task is to answer legal questions and cite retrieved sources. The test offers web research without attached documents, skills or memory.
For questions requiring web search, the reference answers distinguish between essential points and supporting details:
- Essential points: Points needed to answer the question correctly.
- Supporting details: Background or extra explanation included in the reference answer, but not required for a correct, concise answer.
Example: If the question is "What did the court decide?", the court's ruling is essential. Background facts that are not needed to explain that ruling are supporting details. A concise answer can give the ruling correctly without including those details.
The results below measure essential points covered. Supporting details do not affect this score, so omitting optional background does not lower it. Essential points are defined for 97.0% of search-required questions. The remaining questions have no points marked essential and are excluded from this measure.
The ground-truth references also list source URLs. Both configurations cited at least one listed reference source in 24.2% of search-required answers. An answer can still be correct while citing a different valid source.
Reference-data issue. A reference question asks about Hadley v. Western Union Telegraph Co., but a listed reference points to Western Union Telegraph Co. v. Hall. The original U.S. Reports PDF held by the Library of Congress identifies the latter case.
We retained the original scores for both runs. This example is why an automated reference-conflict flag is not a confirmed legal error and why a lawyer should check the underlying authority. We have not independently verified every reference in the dataset.
Both configurations were evaluated against the same dataset and essential legal points. Coverage is measured on questions requiring web search.
Each essential point receives 100% credit if fully covered, 50% if partly covered, and 0% if missing or contradicted. Each answer's score averages this credit across its essential points; the reported result averages those answer scores.
Conciseness: We compare average answer length on the same 15 search-required questions where both configurations covered every essential point in the original grading. This avoids rewarding shorter answers that miss required content. We count words in the final answer, including headings and citation labels, but excluding URLs, reasoning and tool output. Fewer words means a shorter answer on this subset; it does not establish that extra details were unnecessary.
Both configurations used the same additional research model. They ran different numbers of tasks at once, and their runs overlapped, so we do not compare speed.
Both configurations completed 100% of the legal research cases. The evaluator reported no grading failures for questions requiring web search.
-
GPT-6.1 Sol75.7%
-
Opus 5.570.0%
| Measure | GPT-6.1 Sol | Opus 5.5 |
|---|---|---|
| CoverageAvg. essential points covered | 75.7% | 70.0% |
| ConcisenessAvg. number of words | 122.8 | 242.2 |
| Search usedOn questions requiring web search | 97.0% | 100.0% |
| Reference contradictionsAnswers contradicting the reference | 9.1% | 12.1% |
How to use these results when evaluating a vendor
- Test your own work. Select representative matters and include long documents, missing facts, conflicting instructions and material exceptions. Ask the lawyers responsible for the work to grade the results.
- Measure the complete workflow. Include source verification, missed issues, editing time, review effort and whether the final deliverable meets your standards. Test actual accepted Word redlines separately from chat answers or proposals.
- Confirm implementation requirements. Identify primary and auxiliary models, document formats, permissions, matter isolation, retention and approval steps. Reassess after material model or workflow changes.
- Keep the decision with people. Record evidence, unknowns and approval requirements in the approved-tool assessment worksheet. A benchmark score does not approve a tool for client work.
For a practical source-checking exercise, see the worked hallucination example. Use the free Word AI policy template and approved-tool worksheet to set your firm's review rules. Explore Gist's workflows or book a demo to discuss a pilot.
More from Gist
- Free AI Policy Template for Law Firms
- How to Avoid AI Hallucinations in Legal Work
- Generative AI vs Agentic AI for Legal Work
- Will AI Replace Lawyers?
- Which AI Tools Review Contracts Inside Microsoft Word?
- Can Legal AI Draft Using My Firm's Own Precedents and Templates?
- Who Builds and Maintains Custom AI Agents for Law Firms?
