A New York lawyer cited six court cases in his legal brief. Every one was fictitious, generated by ChatGPT, and he became the example everyone now uses in AI cautionary talks.
He wasn’t the last. A Utah lawyer followed in 2025, then a Massachusetts attorney the same year, then a case in British Columbia where opposing counsel caught fabricated citations. These weren’t incompetent people. They were professionals who trusted a tool that sounded confident but wasn’t correct.
I build AI systems for a living. I use them constantly. I also verify constantly, because I have seen how convincing a wrong answer can feel when it arrives in the right format.
The same failure mode shows up in less dramatic places every day: project plans, technical summaries, market research, code reviews, source lists, customer emails.
Part 1 argued that the premium on judgment is higher than ever. But judgment requires a working bullshit detector, and AI is eroding ours.
The Confidence Problem
AI doesn’t know facts. It knows patterns, predicting the most probable next word based on what it learned rather than because it understands truth.
The result is grammatically perfect, contextually appropriate, completely false information delivered with absolute confidence.
How often does this happen? Research on hallucination rates (opens in new tab) shows:
| Model Tier | Hallucination Rate |
|---|---|
| Best-optimized models | 0.7% - 0.9% |
| Average across all models | ~9.2% |
| Complex/specialized domains | 5% - 30%+ |
For legal queries, hallucination rates reach 18.7%. Medical questions: 15.6%.
Here’s the counterintuitive finding: OpenAI’s most advanced reasoning models (opens in new tab) (o3, o4-mini) showed 33-48% hallucination on certain benchmarks, reaching 79% on others. More sophisticated “thinking” doesn’t guarantee more truth (opens in new tab).
I’ve seen this in my own systems. Zeitgaist (opens in new tab), a cross-lingual social intelligence platform I built, had a subtle failure mode I didn’t catch for weeks: queries returned results where search terms matched semantically, but missed contradictory perspectives using different vocabulary. The answer sounded complete. It wasn’t. That’s why I now use cross-checking agents, not to trust AI verifying AI, but to surface disagreements that warrant human review.
The Real-World Damage
This isn’t theoretical. The legal failures above are only the public, embarrassing version of a broader pattern: AI output moves fastest in the places where people are already overloaded, under time pressure, and tempted to accept a fluent answer because it looks complete.
In enterprise settings, the damage is often quieter than a court sanction. It shows up as bad summaries, unsupported market assumptions, wrong source lists, overconfident technical recommendations, and customer-facing responses that someone has to unwind later.
Knowledge workers now spend part of the promised efficiency gain fact-checking AI outputs. That’s time spent verifying whether the machine told the truth.
39% of AI-powered customer service bots (opens in new tab) were pulled back or reworked in 2024 due to hallucination-related errors. The majority of enterprises now include human-in-the-loop processes specifically to catch AI mistakes before deployment.
The efficiency gains from AI are real. But so is the verification tax.
The Content Flood
In November 2024, something crossed over: AI-generated articles published online surpassed human-written articles (opens in new tab) in Graphite’s article corpus for the first time.
The trajectory tells the story. In that analysis, AI-generated articles grew from a small minority before ChatGPT to more than half of newly published articles in 2025. That is not the same as saying half the entire internet is AI-generated, but it is enough to change the default assumption: text online is no longer evidence that a human understood the topic.
There’s a theory called “dead internet,” the idea that most of what you encounter online is now created by AI and bots, not humans. AI generates posts, AI generates engagement, and bots comment on bot content. Nearly half of all internet traffic (opens in new tab) was already bots, according to Imperva’s Bad Bot Report.
The “shrimp Jesus” images on Facebook are the example I cannot unsee: AI-generated devotional pictures attracting huge engagement from accounts that often look automated or low-quality. It sounds absurd, but that is exactly why it works as a warning. Synthetic content does not need to be believable to shape what people think is popular.
The Cognitive Trap
Here’s where it gets uncomfortable. Michael Gerlich’s 2025 study (opens in new tab), published in Societies, found a significant negative correlation between frequent AI tool usage and critical thinking abilities. The more people used AI, the worse their critical thinking became, with younger users showing the sharpest decline. The generation growing up with AI may be the least equipped to evaluate it.
MIT Media Lab published “Your Brain on ChatGPT: Accumulation of Cognitive Debt” (opens in new tab). EDUCAUSE called it “The Paradox of AI Assistance: Better Results, Worse Thinking” (opens in new tab).
The causal picture is still open, but the pattern is enough to worry about: outsourcing thinking to AI is not free. If you stop practicing judgment, you should expect judgment to get weaker.
86% of students (opens in new tab) used AI in the 2024-25 school year, but fewer than half (opens in new tab) worry about skill erosion. They’re using AI to bypass learning the critical thinking skills they need to evaluate AI.
It’s a trap that feeds itself.
The Competitive Divide
The organizational version of this problem is just as ugly:
- Only 5% of companies (opens in new tab) are “future-built” for AI at scale (BCG)
- Only 1% have reached AI maturity (opens in new tab) (McKinsey)
Most organizations lack the critical thinking infrastructure to use AI effectively. They have tools, licenses, pilots, and dashboards. What they often do not have is a habit of asking, “How would we know if this answer is wrong?”
That question separates useful AI adoption from expensive theater.
Individuals have the same problem at smaller scale. The dangerous user is not the beginner who knows they are confused. It is the competent person in a hurry who accepts a fluent answer because it fits what they hoped was true.
What Actually Can’t Be Automated
The World Economic Forum’s 2025 Future of Jobs Report (opens in new tab) lists the most sought-after skills. Analytical thinking tops the list (7 in 10 companies consider it essential). Creative thinking follows. Then resilience, curiosity, and lifelong learning.
Notice what’s missing: “ability to use ChatGPT.”
The skills that remain valuable share a common thread: they involve judgment, not just execution. Analytical thinking matters because someone has to notice when the answer is unsupported. Creativity matters because generating options is cheap, but choosing a useful direction is not. Resilience and curiosity matter because the tools keep changing, and the first answer is often not the one you should trust.
But one skill underlies all of them: the ability to evaluate whether something is true, useful, or right.
That’s critical thinking. And it’s eroding precisely when we need it most.
The Stakes Beyond Your Career
The stakes extend beyond professional success. They reach citizenship itself.
AI algorithms curate your information diet based on engagement, not truth. Filter bubbles restrict what you see. Echo chambers reinforce what you already believe.
Disinformation campaigns have used nearly a thousand bot accounts (opens in new tab) to spread content reaching millions. After major news events, bot-generated posts heavily shape public discussion. Any “overall sentiment” you perceive could be manufactured.
When most citizens can’t distinguish real consensus from synthetic consensus, democracy has a problem. Informed voting requires informed voters. Critical thinking is the foundation.
What to Actually Do
My checklist is boring, which is why it works.
I click links. AI confidently cites papers that do not exist, articles that were never written, and quotes that were never said. If I cannot verify that a source exists, I do not use it.
I get more suspicious when the answer sounds too certain. Expressed uncertainty is often more trustworthy than performed confidence.
For code, I use test-driven development and separate cross-checking agents as disagreement detectors. I do not trust AI verifying AI. I use the disagreement to find the places where I need to slow down and look myself.
Beyond my specific habits: default to skepticism in domains where you have expertise. The more you know, the more mistakes you can catch. Verify high-stakes claims before acting. The time knowledge workers spend fact-checking is worth it, because the cost of acting on false information exceeds the cost of verification.
And question everything synthetic: sources, motives, accuracy, especially consensus that appears suddenly or universally.
The Skill That Remains
Every other skill faces some version of inflation. Coding, writing, analysis, and research all have passable AI versions now. But AI cannot reliably evaluate its own output, cannot know when it is wrong, and cannot exercise judgment about what matters. That’s still on you.
The skill that defines who thrives in the AI age isn’t using AI, because any intern can prompt ChatGPT. It’s knowing when not to trust it. That requires domain knowledge to recognize errors, intellectual honesty to question convenient answers, and the discipline to verify when verification is inconvenient.
Critical thinking was always valuable. Now it’s essential.
The lawyers who trusted ChatGPT learned this the hard way. The enterprises making decisions on hallucinated data are learning it now. The students outsourcing their thinking will learn it eventually.
Better to learn it on your terms.
This is Part 2 of a two-part series. Part 1: The Skill Inflation Paradox covers what’s happening to skills and careers in the AI era.