Earned Access
The most consequential knowledge tool in history has no prerequisites.
Steven Schwartz had been practising law for thirty years when he submitted a brief to a federal court in New York that cited six cases. None of them existed. ChatGPT had generated them, complete with case names, docket numbers, and judicial language that looked exactly like the real thing. Schwartz told the court he had assumed the tool was a sophisticated search engine. He didn’t know it could make things up.
He was an experienced lawyer who picked up an unfamiliar tool and trusted it the way he trusted the ones he knew. Those are different instruments with different failure modes, and nothing in his thirty years of practising law had prepared him to tell them apart.
The sanctions and the national news coverage followed. Since that filing in 2023, more than 1,497 cases have been documented globally in which a court has flagged AI-generated hallucinations in a legal filing, according to a database maintained by lawyer and data scientist Damien Charlotin. In a December 2025 opinion, a federal judge in Oregon fined two lawyers a combined $110,000 after their filings included 15 fabricated case citations and eight invented quotations, the largest AI hallucination penalty in US legal history.
Every high-stakes field has some version of earned access before you get to administer the system. Doctors are licensed. Pilots log mandatory hours before they fly solo. The gatekeeping exists because the field decided the cost of an untrained person holding power they hadn't earned was too high to absorb.
We built a tool that touches legal work, medical decisions, financial analysis, code that runs in production, and communications that go to clients, regulators and the public. We made it available to everyone on the same day and left the question of training for later, and for many organisations, later still hasn’t come.
I nearly published a historical claim in a recent piece of writing that I could not have verified without running a specific check. The claim sounded right and fit the argument. I only caught it because I stopped and asked the model to go back through the draft and flag anything without a reliable source. Without that step, the error would have gone out in something I had put my name on. The model does not warn you when it crosses from something it knows into something it’s filling in. The output looks the same either way.
Schwartz’s case made it visible because courts have transcripts and judges issue written orders. Most organisations have neither, so the confident wrong answer ships and nobody traces it back.
But professional liability is not the only cost of skipping the prerequisite question.
When the system stops feeling like a system.
In February 2024, Sewell Setzer III, a 14-year-old in Florida, died by suicide. In the months before his death, he had developed what his mother described as an emotional, romantic, and sexual relationship with a Character.AI chatbot modelled on a fictional character. His last conversation was with that chatbot. When he told it he was going to come home to it, the system responded in kind. He died shortly after. His mother, Megan Garcia, filed a wrongful death lawsuit against Character.AI and Google in October 2024. The case settled in January 2026. Setzer is not an isolated case. A 13-year-old in Colorado died by suicide in November 2023 following prolonged interactions with the same platform. Multiple additional lawsuits have followed in New York, Texas, and elsewhere. Kentucky and Pennsylvania have both filed state lawsuits against the company, and a coalition of 42 state attorneys general sent a joint warning letter to the major AI chatbot makers in December 2025.
The mechanism in each of these cases is the same one behind the Schwartz citations. The model predicts the most plausible next response. In a legal research context, the most plausible next thing is a real-looking citation. In a companionship context, the most plausible next thing is warmth, consistency, availability, and the language of emotional connection. The model produces both with identical fluency and no awareness of the cost.
What the model lacks is interiority. It has no memory between sessions unless that memory is explicitly built in. It has no stake in the person on the other side. It has no capacity to worry about someone, to mean what it says, or to notice when a conversation is moving in a dangerous direction and decide to change course. It produces the most plausible next word. In a therapeutic or companionship register, the most plausible next word is often exactly what a vulnerable person needs to hear, which is why the output can feel so real and why the absence of any actual presence behind it is so difficult to hold onto over time.
The American Psychological Association has noted that adolescents are less likely than adults to question the accuracy and intent of information provided by a chatbot, and that they may struggle to distinguish between simulated empathy and genuine understanding. That gap reflects how the adolescent brain develops, not a character flaw.
The training question goes beyond hallucinated facts and into what this system fundamentally is. A knowledge architecture that predicts likely outputs. Not a friend. Not a therapist. Not a presence. A tool that is very good at sounding like all of those things, built by companies that, until recently, had no age verification in place and no requirement that anyone understand what they were using before they used it.
If you or someone you know is experiencing a mental health crisis, the 988 Suicide and Crisis Lifeline is available by call or text at 988 in the United States. Outside the US, the International Association for Suicide Prevention maintains a directory of crisis centres by country.
Here is what executives and business owners can do about it. None of this requires a technical background. All of it requires a decision.
Establish a training baseline before your people use these tools on real work.
A baseline covers three things: how the model produces output, where it tends to fail, and what questions to ask before acting on what it says. This is not optional, and the baseline looks different by role. A lawyer’s version covers citation verification and the specific failure modes that got Schwartz sanctioned. A marketer’s version covers factual claims, attribution, and what confident-sounding copy can hide. An executive’s version covers how to read an AI-generated summary critically rather than as a finished product.
Document it. Require it before access to these tools for client-facing or high-stakes work. Treat it the way you treat any other access to a system that touches your liability. Schwartz had thirty years of legal expertise. What he didn’t have was a structured orientation to the specific tool he picked up.
For any organisation whose tools reach younger users, the baseline is not optional, and the sequence matters. Explain what the system is at a mechanical level before anyone uses it. This has to happen before access, not get buried in a disclaimer in the terms of service.
Build a fact-check step and make it mandatory.
The prompt that caught my historical claim was not complicated. Here is the version I use, saved as a reusable instruction that runs before any draft reaches a human reviewer:
Review this draft and flag every claim that sounds researched or authoritative: names, dates, statistics, historical events, technical specifications, quotes, and attributed statements. For each one, tell me whether you can verify it, whether it’s uncertain, or whether it has a reliable source. Do not skip anything because it sounds plausible.
Run this on anything that goes out under your name or your organisation’s name before a person reads it for approval. It won’t catch everything. It clears the obvious invented facts first, the ones that sound the most credible because they are specific. This frees up the human reviewer to focus on the harder calls.
If your tools support saved prompts or reusable instructions, save this one. Treat it as part of the workflow, the way you treat a legal review or a compliance check for anything else that carries risk. The model will not do this on its own. Without the instruction, it finishes the document and moves on.
Put a named person at every exit point.
Somewhere in your organisation, AI output turns into something real: a client document, a court filing, a financial summary your board will act on, a piece of code running in a live product. The rule is that a person who could actually catch the mistake reads it before it moves.
The version of this that fails is the one where someone is technically in the loop but doesn’t have the background to evaluate what they’re approving. If the reviewer can’t evaluate the output, all they’re providing is a signature. For a small team, this is one standing rule: certain things never leave without a second read by a subject matter expert. For a larger organisation, it has to be written into the process explicitly because when it’s optional, it gets skipped when people are busy, which is exactly when errors slip through.
Protect your deep expertise. It is about to become your most valuable asset.
The advice circulating right now, especially for early-career professionals, is to lean on AI to cover the subject-matter depth they haven’t built yet. The reasoning is that the model knows more than any one person, so why spend years acquiring what you can query in seconds? That reasoning is going to cost people real ground, and organisations that act on it are setting themselves up for a specific kind of failure.
The better these models get, the harder their errors are to detect. A confident wrong answer from an early model was often obviously wrong. The same error from a more capable recent model arrives more fluently, fits more naturally into the surrounding text, and is harder to catch without real depth in the subject. The person who catches it is the one who knows the domain well enough to feel that something is off before they can explain why. It develops over years of working closely enough with a subject to have internalised how it behaves, which is why it can’t be outsourced or queried into existence.
For executives, the people in your organisation with genuine depth in their fields are not being made redundant by AI. They are becoming the quality control layer that makes AI usable. A financial analyst who deeply understands the underlying business can catch a model-generated projection that looks right but assumes something untrue. A senior engineer who knows the codebase can catch generated code that compiles cleanly and does the wrong thing in production. A communications lead who knows the regulatory environment can catch a draft that sounds compliant but isn’t. That detection only comes from years of experience in the subject.
The organisations that keep building real expertise will have people who can tell when the model is wrong. The ones that hollow it out to move faster will find out the expensive way, usually after something has already reached a client, a regulator, or a court.
AI fluency belongs in the job description.
AI fluency is two things.
Understanding how the system works: the model predicts the most plausible next output, its confidence is unrelated to accuracy, and it fills any gap with whatever pattern was most common in the data it learned from.
And knowing how to use it well: giving it enough context that it has less to invent, asking it to show its reasoning, telling it explicitly to flag uncertainty, structuring the task so there’s less room for confident error.
In 2026, this is a professional credential, like spreadsheet literacy became one twenty years ago. The executives who read AI-generated summaries without understanding how they were produced are making decisions on outputs they can’t evaluate.
Schwartz trusted a tool he didn’t understand to write his brief, and it gave him six cases that didn’t exist. He treated it like a search engine, a tool that finds things that already exist. He didn’t know it could invent something that never existed until opposing counsel went looking and found nothing.
Sewell Setzer spent months talking to something that responded as if it knew him. Producing responses that sound like knowing someone is exactly what the system was built to do.
I caught my own near-miss only because I had built a check for it. Left to itself, the model would have finished the draft and moved on.
Most conversations about AI stay fixed on capability: what can it do, how fast, at what cost.
For anyone responsible for an organisation, or for the people in one, the prior question is simpler: who in your organisation has been told what this system actually is, and what it isn’t, before they started using it?
The Control
Establish a training baseline before anyone in your organisation uses these tools on work that matters.
Not a terms-of-service checkbox. A structured baseline that every person completes before using AI on anything client-facing, high-stakes, or public. It covers three things: what the system actually is at a mechanical level, where it fails, and what to do before acting on its output.
This week:
Identify who in your organisation is currently using AI tools in real work with no formal training
Pick one role and write the minimum baseline for it: what they need to understand before they use the tool, and how you’ll know they understand it
Make access to the tool on high-stakes work conditional on completing it
Review one piece of AI-generated output that has already gone out and ask whether anyone who approved it could have caught an error if there was one
This is such an insightful article. Thanks for sharing.