How to check AI’s work

The problem with AI is not that it gets things wrong. Everything gets things wrong, including me. The problem is that it sounds exactly as sure of itself when it is wrong as when it is right, so the usual signal you rely on, that little wobble in someone’s voice when they are guessing, is just not there.

A couple of studies put real numbers on this, and they are worth knowing before you trust anything on a first read.

The Tow Center for Digital Journalism at Columbia ran 1,600 queries through eight AI search tools in 2025 and found they gave incorrect citations more than 60% of the time, ranging from 37% for the best one to 94% for the worst. The part that stuck with me: in their ChatGPT tests it got 134 out of 200 article attributions wrong, and in all of those it signaled any uncertainty only fifteen times and never once declined to answer. They also found the paid tiers were more confidently wrong than the free ones.

Stanford’s RegLab did something similar with law, published in the Journal of Legal Analysis in 2024. They asked models specific, checkable questions about randomly chosen federal court cases, and depending on the model, between 58% and 88% of the answers contained something made up. They also found that when the question itself contained a false assumption, the models often just went along with it instead of correcting the person asking.

Those are hard cases, law and news attribution, and the tools have gotten better since. That is not really the point. The point is that none of that showed up in the tone. It read like an answer, every time.

In the hospital we call this escalation, and it is a trained reflex more than a rule. You learn very early that the strongest thing you can say is not the right answer, it is "I don’t know, I’m getting help." A good nurse knows exactly where the edge of their own knowledge is, and when they hit it, they stop and they call someone.

AI does the opposite of that. It does not have an edge it can feel, so it never stops, and it never calls anybody. It answers. Every single time, no matter how far outside its knowledge the question sits.

So the whole job here comes down to one sentence: if the AI is not going to escalate, you have to be the escalation. You are the last set of eyes before this goes out into the world with your name attached to it. Everything below is just the practical version of that.

One thing worth saying plainly, because it saves you most of this work: a lot of what looks like AI being wrong is actually AI being under-briefed. If you want fewer things to catch on the back end, put more in on the front end. That is what the SBAR guide is about.

This is the first thing everyone tells you and it is genuinely good advice, so start here: for anything factual, ask it to give you its sources with direct links.

Here’s the deal though, and this is the part that usually gets left off. Asking for citations does not fix the problem on its own, because the citation is text too, and it can be invented right along with everything else. Researchers looking at AI-assisted academic writing have found fabricated references at genuinely uncomfortable rates: real-sounding titles, plausible author names, journals that exist, article IDs that lead nowhere. Connecting the model to a search tool helps but does not close the gap either, since it can still garble what it found on the way back to you.

So the instruction is not "ask for sources." The instruction is ask for sources, open them, and confirm the page actually says the thing. That last step is the whole check, and it takes about forty seconds.

Two tells I have learned to watch for. If it hands you a source with no link, treat that as unverified until you go find it yourself. If it hands you a link to a homepage or a search results page rather than the specific article, that is usually a sign it knows the publication exists but not that the specific claim does.

Checking whether something is true is only half of it. The other half is whether it is yours, and that is a separate question with a separate answer.

The rule I set for myself when I started using AI for writing is that it is allowed to help me draft, and it is never allowed to sand me down. It can get the shape of a thing onto the page much faster than I can. What it cannot do is know which sentence is the one that actually matters to me, and left alone it will quietly smooth that sentence into something that sounds fine and means nothing.

What that looks like in practice is small. Find the one line that carries the point, and rewrite that line yourself, in your own words, even if the version on the screen is technically better written. Cut the phrases you would never say out loud. If something in there is a real opinion, make sure it is actually your opinion and not just the most agreeable version of one.

People can tell. Not always consciously, but they can tell, and the trust you lose there is much harder to get back than a fact you got wrong and corrected.

None of this makes AI safe for decisions where being wrong is expensive. I am a nurse and I will say this as plainly as I can: nothing here is medical advice, and I would not let an AI answer decide anything medical, legal, or financial for me. Checking its work lowers your error rate. It does not move the responsibility off you, which is the entire point of being the endpoint.

Running the same question past several different AI models and comparing them is a real technique and there is good research behind it, but it is a bigger topic than a paragraph at the bottom of a guide, so I am going to cover it on its own.

Not everything needs this. I am not auditing a recipe or a first draft of an email to a friend. I run these checks when the answer is going somewhere public, somewhere permanent, or in front of someone whose opinion of me I care about. Most of what I do with AI is still just typing a question and getting on with my day.