What Legal Teams Can Learn from Sullivan & Cromwell’s AI Hallucination Crisis
Want more content like this? Sign up for our monthly newsletter.
Sullivan & Cromwell, one of Wall Street’s most prestigious law firms, recently apologized for submitting a court filing riddled with errors.
Mistakes happen. But they don’t usually make it to the New York Times.
What made this particularly news-worthy is that these errors were the result of AI hallucinations, a term referring to the way artificial intelligence can make up facts and figures without careful review. These errors, found by the opposing counsel, were filed in an emergency motion in a Chapter 15 bankruptcy proceeding in the U.S. Bankruptcy Court in Manhattan.
It included approximately forty AI-generated errors including fabricated case citations, incorrect pin cites, misquoted authorities, and references to cases that didn’t contain the quoted language. The apology letter to Chief Judge Martin Glenn required a three-page, single-spaced attachment just to list them all.
Oof.
According to the apology letter, Sullivan & Cromwell has clearly stated policies around the ethical use of AI, and that they were not followed in the filing. This is exactly why it’s important to make sure there’s always a human in the loop whenever you’re working with AI, but especially in high-stakes situations that require careful decision-making and attention to detail. In this post, we’ll talk more about why these errors keep happening—and what legal teams can do about them to make sure they use AI responsibly.
AI hallucinations continue to be a top concern for legal professionals
These kinds of errors aren’t new. Legal tech researcher Damien Charlotin has catalogued over 1,300 documented AI hallucination cases in legal filings.
In 2023, a federal judge in Manhattan fined two lawyers $5,000 after they submitted made-up case filings, according to the New York Times. Another in Mississippi punished four lawyers with $8,000 in fines and canceled a civil trial over AI use. “In some earlier examples of the same behavior, offending attorneys were let off with a slap on the wrist,” University at Buffalo School of Law professor Mark Bartholomew told Business Insider. “Now judges just don’t buy it when a lawyer protests they had no idea that AI models can hallucinate, and they are willing to call such behavior out as bad faith.”
When a mistake like this happens, who is really to blame? The ABA’s Formal Opinion 512 established individual lawyers’ duty of competence over AI tools they use, but left accountability for outputs largely at the practitioner level. Most of our respondents agreed that when AI fails, it’s a failure of policy and technology, not the individual.
Source: 2026 State of AI in Legal
This is exactly what legal professionals across corporate teams and firms are worried about. 92% of legal professionals we surveyed for our State of AI in Legal Report use AI at work. But accuracy concerns remain a top issue.
High-profile cases like this come to mind, but there’s also more day-to-day risk: If there’s a mistake buried in your contract, it could take years before someone spots it, opening up your organization to risk, costs, or a breach of trust with the other party.
Why AI hallucinations happen
There are several reasons that this occurs from a technological perspective. It can be deceptively easy to picture actual intelligence behind generative AI’s cheerful, conversational outputs, but the reality is that AI works by analyzing language patterns that go together around a specific topic to create a response. There’s no “thinking” beyond parsing words, phrases, and sentences into a logical pattern.
That’s why it can be difficult to detect when AI has made a mistake. It all seems fine on the surface, but without careful fact-checking, you’ll run into the same errors that Sullivan & Cromwell did.
The other reason this often happens is related to data. The phrase “garbage in, garbage out” applies here, and it’s exactly why it’s important to train your AI models on your specific organization’s context—and why AI companies like Anthropic, Microsoft, and OpenAI have all come out with legal-specific plug-ins or add-ons for their popular generative products.
The real problem lies in the process
The benefits of artificial intelligence—efficiency, collaboration, and time-savings—are real. But they can’t make up for the machinery in an organization that puts intense pressure on their legal teams to work faster without any support.
If anything, adding AI has only added to the workload.
Increased expectations, coupled with a process that requires stitching together multiple platforms or constantly running into administrative black holes, makes it easy for errors to take place. If you’re under pressure to get something out the door yesterday, do you really have time to do a third or fourth round of review where you’d catch those clerical or AI errors?
Because AI output “seems” correct, it can be easy to gloss over small details under pressure. A policy may be in place, but if your organizational process creates negative incentives for your team to get it done, no matter what—these errors are going to keep happening.
How in-house teams can set their people up for success with AI
First, if you don’t already have one: Create a clear policy on how to use AI, and make sure your entire team is on board. Only half of teams we surveyed had a clear policy in place, so this should be #1 on your to-do list. You can’t enforce a hand-wavey “it’s fine to use it.”
Understand your current AI usage
Work through your most common output types as a team: Contract summaries, first-pass redlines, research memos, risk flags, stakeholder communications, final recommendations. For each one, agree on whether AI output can be acted on directly, needs one person to review it, or needs senior sign-off. All you need is one hour and a shared document. The idea is to make sure it’s clear when a human should be involved, when management should be involved, and when it’s okay to let AI do its thing.
Define tasks AI does best
When we asked legal professionals which tasks worked best for AI outsourcing, contracts came up again and again. Summarizing contracts, tagging contract metadata, and other menial tasks are a great way to speed up your workflow without introducing unwanted errors. Your policy should clarify how much or how little intervention is needed for gray areas like conducting legal research, drafting legal memos, or flagging and replacing contract clauses.
Finally, build automated guardrails into your workflow
The difference between an AI environment that produces smart, easy-to-use outputs and one that often creates hallucinations comes down to the system that you use, not the specific AI model. You’re going to want to use a model that is specifically trained on legal information, and ideally one that is part of a larger system that understands how contracts and case law get made.
“The only way that AI can improve over time is to have your organization’s context, the rules and standards by which you operate,” says Suha Saya, Director of PMM, Jurist, here at Ironclad. “A regular AI tool is just using the internet and whatever prompt you put in, but when it’s embedded into a system that’s already designed for legal specialists, you can get a much more customized answer for a legal question and more tailored recommendations for your deal playbooks, for example. That kind of thing would take months to do without the system in place.”
The data quality, workflow design, and system context all matter for how it performs for your legal team.
Don’t let AI hallucinations happen to your team
The good news is that with the right data and feedback, you can train AI out of hallucinations, especially when it’s wrapped into a larger piece of software that provides guardrails on the outputs. “Models are trained on sets of data, and each round helps cross-check all of its references. Hallucinations are decreasing across all model providers, and I don’t think it will be a problem in a year or two,” says Blaine Bassett, former Product Marketing Manager at Ironclad.
Overall, most teams believe that the benefits of using AI outweigh the risks of errors and hallucinations, especially as the technology continues to improve.
For actionable advice on how to create your own robust accountability policies, check out our recommendations.
Ironclad is not a law firm, and this post does not constitute or contain legal advice. To evaluate the accuracy, sufficiency, or reliability of the ideas and guidance reflected here, or the applicability of these materials to your business, you should consult with a licensed attorney.
