Why is it that some people can leave AI to work overnight and wake up to incredible results -- while others struggle to get anything usable out of constant, iterative prompting?
Andrej Karpathy wrote about this on X recently. For some people, AI still "looks like a derpy chatbot." Meanwhile, “a small sliver of professionals are experiencing the vertigo of the curve going vertical.”
So what explains the gap? He points to a few things, but I found this part of his answer interesting:
"Capability peaks in domains that are digital, verifiable and economically valuable."
The word in that sentence I'd underline is verifiable. “Verifiable”, in the context of what I want to chat about today, just means the AI can check whether its work is right.
When AI writes code, for example, it can run that code right away to see if it works. If something is wrong, it fixes it and runs it again. It can keep going for hours like that because whether the code successfully runs or not is a form of verification.
Note: As a developer, I recognize that most code is not completely verifiable — just because it runs and somewhat works in a given situation doesn’t mean it’s right. But, relatively speaking, code is more verifiable than other outputs because it has a defined purpose. If a piece of code is supposed to process a transaction, update the customer record, write a transaction log and send a message to the customer, we can test to see of those things happened after the code is run.
Anyhoo, back to the matter at hand…
If you’ve ever received an enthusiastic “job done!” type of response back from your AI agent that turned out to be disappointingly premature or wildly optimistic, you’ve experienced what happens when AI has no way to verify its work.
So today, I want to break down:
What verification is, and how it could 2-3x the quality of your results
The three kinds of checks you can use for verification (with example prompts)
Why verification isn’t bulletproof (but can still save you time)
Looks Done vs. Is Done
Anthropic's guide to Claude Code describes the verification problem better than I could: "Claude stops when the work looks done. If it has nothing to run that tells it whether the work actually is done, 'looks done' is all it has to go on. In practice that makes you the test suite."
When the AI has no way to check its own work, the checking lands on you.
Boris Cherny, the creator of Claude Code, said back in January that this is "probably the most important thing to get great results out of Claude Code -- give Claude a way to verify its work. If Claude has that feedback loop, it will 2-3x the quality of the final result."
That 2-3x is his own estimate, not a lab measurement. But it's coming from the person who built the thing, so I take it seriously.
Why does verification matter so much?
Without verification, the AI is left guessing at what “finished” looks like, constantly starting and stopping out of a lack of clarity.
This is how agentic engineers have scaled their productivity to incredible levels with AI. With the right verification systems in place, you can hand a frontier model a goal that it will not stop working on until achieved.
So how can you get started with verification?
Three Kinds of Simple Checks
You don’t need to be an engineer to start using verification.
The pros build elaborate test systems, but the basic idea works in any ChatGPT or Claude chat. You just ask the AI to check its own work before it tells you it’s done.
Not every way of checking is equally reliable, though. It helps to know the difference between a grade and a check. (These are simplified terms, not official terms. I’m a simple guy)
📋 A grade is the AI’s opinion of its own work. Ask “Is this email clear and persuasive?” and you’ll get something like “Yes, 9 out of 10.”
✅ A check comes back pass or fail, no matter how the AI feels about its work. “Is this email under 150 words?” has exactly one right answer, and the AI can’t talk its way into a pass.
The difference is who you’re trusting. With a grade, you’re trusting the AI’s judgment of its own work, and the same AI that made a mistake is the one deciding whether the work is good. With a check, you’re trusting something you could confirm yourself.
The less a check depends on the AI’s judgment, the more you can trust it. Here are three kinds of checks that you may find useful in your own work:
✅ 1. Count it. Rules with exactly one right answer: “under $500 per person”, “under 150 words”, “nothing before 10 a.m. on Saturday”.
The work meets the rule or it doesn't. That also means you can check the AI's answer yourself in seconds. Use checks like this wherever you can -- they’re the most reliable.
✅ 2. Look it up. Facts the AI confirms by checking a secondary source.
Tell the AI where to look: a website, or even just “Write the landing page copy using this PDF of our feature list as your only source of truth.” Then have it cite the exact page from the PDF it used to write each section.
✅ 3. Try it. The AI walks through the result the way a real person would.
Back to Boris Cherny -- he has Claude open a browser and test its own change: it "opens a browser, tests the UI, and iterates until the code works and the UX feels good.”
Some things you can’t count or look up. The only way to know if they work is to try it. So you ask the AI to do exactly that. If you try this kind of check, make sure to include something that could actually fail, like a button that has to work.
Like this: “Inspect every button on the page and tell me which ones are missing links to the landing page.” -- Not this: “Review my landing page”.
When we combine all three of these checks, here's what a research prompt becomes:
Help me pick a CRM tool for a team of [5] under $[X] per person per month. Follow these rules in your research:
COUNT IT: For each option, mark pass or fail: under budget at our team size; works with [tools we use]; has a free trial.
LOOK IT UP: Confirm current pricing and integrations from each vendor’s own pricing and integrations pages, with links. If a price is “contact sales,” note it as such in your final report.
TRY IT: Using each vendor’s own documentation, write a short onboarding SOP for our team, with a link to the real supporting doc for each key step.
And another, for building a landing page based on a screenshot:
Here’s a screenshot of the landing page I want [attach screenshot]. Build it. Before you tell me it’s done, check your work three ways:
COUNT IT: Mark pass or fail: every section in the mockup is on the page, in the same order; the headline and button text match word for word; the page works at phone width.
LOOK IT UP: Take a screenshot of your page and put it next to my mockup. List every difference you see: spacing, colors, fonts, missing images.
TRY IT: Click every button and link the way a visitor would, on a desktop window and a phone-sized one. Flag anything that does nothing or goes to the wrong place.
Fix any errors found in the testing process, take a new screenshot, and show it to me next to the mockup.
These prompts are interesting because we’re not instructing the AI to follow a detailed step-by-step process to create the outcome. Instead we’re creating rules around the outcome to raise the floor on the range of outcomes the AI will create for us.
This is subtle but important. As the AI models get better and better, you want to provide fewer and fewer instructions about the “how” and more about the “what” a successful output looks like.
Tell the LLM about the destination you’re going to, not turn-by-turn directions on how to get there.
By giving the model freedom to choose its own path we avoid bottlenecking it. And by equipping it with the tools and information it needs to verify that the desired outcome has been achieved, we give it the guardrails it needs to work until the goal is sufficiently achieved.
If you’re building with AI, verification is where the leverage is. Empowering your agents to check their own work means you don’t have to sit around and babysit a chat bot. Instead, you’re free to design more agentic loops. The folks you read about who leave AI agents running for hours and come back to finished work are almost always the ones who’ve gotten good at verification.
Verification Is Powerful. But It’s Not Perfect.
Your verification is only as good as the rules you give it.
A big part of working with AI agents is making sure the agent clearly understands what you want the outcome to look like.
If there’s something you care about that you never wrote down as a check, nothing will catch it. It happens. Especially when you’re working outside your area of expertise.
But the idea is that you will reap greater long-term ROI refining your broad verification processes than you will chatting back and forth refining a specific one-off task.
At the very least, a light layer of verification can help you reduce the amount of time you spend double checking an agent’s outputs.
Even good verification has its limits. Lauren Tan, a SpaceXSI engineer who posts regularly about verification, says the point is “to give yourself confidence in the agents’ work.” The same post is honest about the limits: verification “is just approximation and can still be wrong.”
Today’s frontier models are ready and waiting to take (some of) the burden of verification off our shoulders. If you’re using AI to build, I encourage you to experiment with the ideas we touched on in this post.
Why do some people wake up to finished work while others spend the day prompting? A big part of the answer is verification. The people waking up to finished work gave their AI a way to know when it’s actually done.
So the next time you hand AI a task, ask yourself one question: how will it know the job’s been done right? Because if you don’t have a clear answer, neither does the AI.
—Dharmesh (@dharmesh)
P.S. Speaking of AI that works while you sleep: YouSpot, the AI-native personal CRM I’ve been building has several useful features. First, it lets you chat with all of the top models (because some models may be better than others at your use case). Second, it lets you turn your prompts into cloud agents that are always running and always working for you. Third, it helps you build a “Second Brain” to help you stay on top of what you’re working on. As a Simple AI subscriber, I’m offering 20% off the $10/month price (no long-term commitment). I’m biased, but you should try it.
Here’s the promo link: https://youspot.com/pricing/pro?promo=simpleaifan
I’m building it in public over on LinkedIn if you’d like to follow along.
