Today’s issue is brought to you by

Remember when Google's AI told people to put glue on pizza? Or the lawyers who submitted court cases ChatGPT had completely made up?

We all learned the same lesson: AI makes things up. You have to check everything.

Now we're seeing AI systems beat PhD specialists at checking research claims. And I know some of you will read that, look at the nonsense ChatGPT gave you this morning, and wonder what the hell I'm talking about.

Fair! How can I tell you AI is better at humans at factual accuracy whilst you are still getting crap from it?

Ethan Mollick flagged this up recently. We need to update our assumptions about who is better at finding facts and checking claims.

A good example here is BrowseComp. It tests whether you can track down obscure information online, following clues across different websites. The answer is hard to find, but straightforward to check once you've found it. Human testers answered about a quarter of the attempted questions correctly. OpenAI reports 92.2% for GPT-5.6 Sol Ultra. Blows humans out of the water.

Then there's DeepFact, which tested difficult research claims. On hidden known-answer claims, PhD specialists (humans) initially scored 60.8%. The three agents tested started between 72.7% and 88.8%. With agents auditing each other's work, the best pairings reached 93%.

Agents beat PhD specialists.

To be fair with repeated checks humans improved too: auditing disagreements raised their score to 90.9%. Still below AI but close!

The assumption that humans must always be the better fact-checker is looking pretty shaky…

Andrej Karpathy calls language models "dream machines". They've learned patterns from enormous amounts of material, and they generate a response from those patterns and whatever information you put in front of them.

They're very good at producing something that sounds like a real book title, quotation or court case. I call this the shape of meaning. It sounds good, then you knock on it and find it's hollow.

That was a huge problem for the first couple of years of modern AI - hallucinations made AI too unreliable for day to day usage.

But…it’s sorta solved.

To understand why that is imagine you're comparing software for your business. How much does a CRM cost? You vaguely remember paying $100 a month at your old company. Would you make the buying decision from that?

No. As a human making a decision you wouldn’t base it on what you sorta kinda remember.

Instead you’d open the company's pricing page and check the pricing.

An AI agent can do those steps too.

Fundamentally this has shifted how useful AI is. It can search, open pages, compare the plans and calculate the cost for your business. The information it finds becomes part of the context for its answer, giving it something current to work from.

And when two pages give different prices, it can investigate. Is one old? Are they quoting different plans? Does the current official pricing page explain the difference? It can go round again until it has an answer supported by evidence, or tell you what it couldn't establish.

If you've used Claude Code or Codex, this will be familiar. Write the code, test it, inspect the failure, fix it. Those multi-step tools are now spreading into the rest of our work.

It’s just that most of the world hasn’t realised. They are still thinking of 2022 AI that was, let’s be honest…a bit crap!

OK so how do you make sure you are getting the “most accurate” AI?

You don’t ALWAYS need it. Sometimes speed is more important. But for questions that genuinely need to be correct then we can do a few things.

First, the model. For a complicated job where accuracy matters, use a capable model and give it more reasoning time. Look for the reasoning or effort setting in your tool. A quick-response mode is useful when you want a quick response; a proper investigation needs time to do the work.

This means a paid model by the way. If someone complains about AI not being good but they are on a free plan you can generally ignore them!

Second, access. What can it actually see? If you're asking about your business, much of the information will be private. Give it the relevant documents. If you want it to examine a contract, your half-remembered account of what the contract says is a poor starting point. It needs to see what you are talking about to assist,

This is why I keep coming back to building an AI brain. Keep your business context somewhere your tools can read it.

Third, instructions. Tell it you need research and verification. Asking for a quick answer, then getting annoyed that it hasn't investigated thoroughly, is a communication failure. On your side! Tell it what decision you're making and what needs checking and it’ll act accordingly.

For more help putting this into practice, I've got eight practical AI crash courses for $8.

To the Task,

Kyle