ChatGPT writes like it knows everything with extraordinary confidence. That’s precisely the problem. Similar to the human know-it-alls many of us have encountered, confidence does not equal accuracy…which makes it as dangerous in legal work as it is useful.
Since OpenAI launched ChatGPT in November 2022, attorneys have experimented with AI tools for drafting contracts, summarizing depositions, and generating research outlines. The productivity gains are real. Tasks that once took hours can now take minutes.
The problem is that ChatGPT generates language based on probability and pattern recognition, not verified legal reasoning. The result can be professional-sounding work product containing fabricated citations, incorrect legal analysis, or subtle drafting mistakes that become expensive later.
Courts have already responded. Attorneys have faced sanctions and reputational damage after submitting filings containing nonexistent case law generated by AI. The issue is no longer whether AI belongs in legal practice. It already does.
The real question is whether firms are using it deliberately or simply trusting a tool that sounds more certain than it actually is.
This post breaks down where ChatGPT falls short in legal drafting, the risks firms need to understand, and how attorneys can use AI responsibly without sacrificing accuracy, confidentiality, or professional judgment.
ChatGPT is an AI chatbot built by OpenAI using a large language model (LLM) known as GPT, short for Generative Pre-trained Transformer. It was trained on massive amounts of text pulled from books, websites, articles, and other written sources, allowing it to generate human-like responses to prompts.
Its adoption spread quickly across industries, including legal services. Attorneys began using it to summarize documents, draft emails, outline arguments, and accelerate administrative work.
The appeal is obvious. ChatGPT can generate first drafts in seconds and reduce repetitive writing tasks that often slow legal workflows down.
The problem is that ChatGPT does not retrieve facts the way a legal research platform does. It predicts statistically likely language patterns based on its training data. In legal work, that distinction matters because text can sound authoritative while still being incomplete, inaccurate, or entirely fabricated.
In legal drafting, confidence and correctness are not the same thing.
Law is built on precision. Contracts, motions, briefs, and legal opinions all depend on accurate citations, careful interpretation, and precise language. So when a tool appeared that could generate legal-sounding text instantly, attorneys paid attention.
For firms managing heavy caseloads, the productivity gains were real. Then came the reality check.
Lawyers began submitting filings containing case citations that didn’t exist. Courts issued sanctions. Bar associations started publishing ethics guidance surrounding AI use in legal practice.
The most widely discussed example was Mata v. Avianca, where attorneys filed a brief containing six fictional cases generated by ChatGPT.
The lesson was not that AI has no place in legal work. It was that attorneys cannot delegate professional judgment to systems they do not fully understand or independently verify.
The problem wasn’t using AI; it was using it without understanding its limitations or verifying its output.
Understanding why ChatGPT struggles with legal drafting is essential before using it in professional practice. These are not isolated edge cases or occasional glitches. They are structural limitations tied directly to how large language models work.
When ChatGPT lacks reliable information, it may generate answers that sound plausible rather than acknowledging uncertainty. In legal contexts, that can include fabricated citations, nonexistent statutes, inaccurate legal analysis, or misrepresented precedent.
Research associated with Stanford’s legal AI benchmarking efforts reported error rates ranging from roughly 69% to 88% in certain legal liability-analysis scenarios involving unsupported AI-generated reasoning.
In Johnson v. Dunn (N.D. Ala.), hallucinated citations appearing in the briefing contributed to judicial scrutiny surrounding attorney AI use and verification practices.
Judges are increasingly making the same point: AI errors are still attorney errors.
ChatGPT has exposure to legal terminology, but exposure is not the same as legal reasoning.
The system often struggles with jurisdiction-specific interpretation, nuanced contractual language, and context-dependent legal analysis. Provisions involving liability limitations, enforceability standards, or procedural requirements can easily be misunderstood or oversimplified.
Surface familiarity with legal language is not the same as legal understanding.
Public AI systems may retain prompts for model improvement or system training purposes.
For attorneys operating under strict confidentiality obligations, this creates obvious concerns when client information is entered into unsecured public tools. Samsung’s widely reported internal data exposure incident demonstrated how easily sensitive information can be leaked when organizations adopt AI without clear operational guardrails.
ChatGPT cannot provide legal advice, assume liability, or uphold a professional duty to a client.
When its output is wrong, and sometimes it will be, the attorney relying on that output remains fully responsible for the consequences.
Some of the most expensive legal mistakes are subtle.
ChatGPT frequently overlooks hidden fee structures, auto-renewal clauses, indemnification language, intellectual property ownership provisions, and termination conditions.
Attorneys and mediators have increasingly reported disputes tied to AI-generated agreements missing important contractual language or failing to account for business-specific obligations and risk allocation.
Sometimes the danger is not obvious malpractice-level failure. It is small drafting language that quietly creates major business consequences later.
Responsible AI use starts with reliable information.
In our earlier post, Stop Searching, Start Finding: The New Era of Legal Data, we explored how legal professionals are rethinking access to trustworthy legal information. That naturally leads to the next challenge: knowing when AI-generated output can actually be trusted.
AI has a place in legal practice, but it requires guardrails, oversight, and verification.
Every AI-generated citation should be verified against authoritative databases like Westlaw or LexisNexis before appearing in a filing or client deliverable.
One fabricated citation can undermine an entire argument.
At the 2023 ABA TECHSHOW, Casetext’s Pablo Arredondo summarized a key principle: anchor AI to a source of truth.
AI becomes significantly more reliable when it operates inside controlled workflows tied to verified documents, approved clause libraries, and validated research sources instead of unrestricted prediction.
ChatGPT works best when accuracy depends on information already reviewed by the attorney rather than independent legal analysis.
Appropriate tasks include:
If public AI systems are being used, identifying client information should be removed before documents are uploaded.
That includes party names, financial information, contact details, and confidential business data.
Whenever possible, firms should prioritize legal-specific AI platforms offering stronger confidentiality controls and Zero Data Retention environments.
Purpose-built legal AI platforms such as Spellbook are designed specifically for legal drafting and review workflows.
Many include clause benchmarking tools, structured review systems, secure data environments, and Microsoft Word integrations that general-purpose AI tools often lack.
Every law firm using AI should establish written policies governing approved use cases, verification requirements, confidentiality procedures, and accountability for final review.
As courts continue responding to AI misuse, AI governance is quickly becoming part of professional risk management itself.
Even when firms understand AI’s risks, implementation can create new problems.
Over-reliance on speed. AI produces answers quickly, which creates pressure to accept output without sufficient review. Courts have repeatedly made clear that speed is not a defense for inaccurate filings or unsupported citations.
Treating AI as a finished product. Baker Donelson’s AI team described the issue well: AI output should be treated like the work of a junior associate that still requires supervision and citation checks.
AI drafts are starting points, not final products.
Neglecting client education. Clients increasingly use ChatGPT themselves, sometimes generating contract language before consulting counsel.
Attorneys should explain where AI can assist and where professional legal judgment still matters.
Ignoring evolving court requirements. Courts are continuing to respond to AI misuse. Since 2023, legal researchers and industry trackers have identified well over 100 cases involving alleged AI-generated hallucinations, citation errors, or disclosure disputes tied to legal filings.
Some jurisdictions now require disclosure of AI use in filings, and attorneys must stay current on evolving court rules, ethics opinions, and professional guidance.
Used thoughtfully, AI can deliver real operational advantages over time.
Efficiency improves significantly. AI-assisted workflows can reduce time spent on first drafts, document summaries, and contract review preparation, allowing attorneys to focus more on strategy and client relationships.
Consistency improves as well. AI tools integrated with firm clause libraries and playbooks help standard agreements align with established firm positions and drafting standards.
Client experience can also improve through faster turnaround times and clearer early-stage drafts.
Finally, structured AI use supports stronger risk management. Firms with clear policies, verification protocols, and purpose-built tools are better prepared to adapt as compliance standards and court expectations continue evolving.
The firms seeing the greatest success are not necessarily using AI the most. They are using it most deliberately.
AI is already changing how law firms research, draft, review, and communicate. The firms seeing the greatest long-term benefit are not necessarily the ones adopting AI the fastest, but the ones implementing it thoughtfully with clear guardrails, verification standards, and operational policies that protect accuracy, confidentiality, and client trust.
For law firms, the challenge is no longer deciding whether AI belongs in legal practice. The challenge is learning how to use it responsibly without weakening professional judgment, internal standards, or client protections in the process.
That requires more than software. It requires strategy, governance, security, and operational discipline.
Heroic Technologies helps law firms evaluate AI platforms, secure sensitive data, establish governance policies, and build practical workflows that allow firms to benefit from AI without sacrificing professional responsibility. The goal is not to replace legal expertise with automation. It is to help firms use technology deliberately, securely, and in ways that genuinely support better client outcomes.
AI tools will continue evolving quickly. Professional accountability is not going away alongside them. Connect with Heroic Technologies to start building an AI strategy grounded in security, practicality, and responsible implementation.
1. Can lawyers use ChatGPT legally?
Yes…with caution. Most bar associations permit AI use as long as attorneys follow professional rules on competence, confidentiality, and diligence. Regardless of the tool used, lawyers remain fully responsible for the accuracy and ethics of their work.
2. What's the difference between ChatGPT and legal-specific AI tools like Spellbook?
ChatGPT is a general-purpose language model trained on broad internet data. Legal-specific tools like Spellbook are built for legal drafting, trained on legal text, integrated with platforms like Microsoft Word, and designed with stronger confidentiality safeguards.
3. How should a law firm build an AI policy?
Start by defining what AI can and cannot be used for in your firm. Require verification of any AI-generated content used in client work or filings, and assign clear accountability for final review. Policies should be reviewed regularly as AI tools and court guidance evolve.