Getting the AI to show its work

July 7, 2026

Hand-drawn illustration of three small robots at a desk: one proudly holds up a star-rating card, one scowls through a magnifying glass at a document, and one in a judge's wig raises a gavel — while a bearded guy in a backwards cap and headphones watches with arms crossed.

The problem with "vibe coding"

Most AI features these days are what people call vibe-coded: you write a prompt, the model says something confident, and you ship it. It feels like magic right up until it's wrong. And when it's wrong, it's wrong in a very confident way. AI folks call this hallucinating, where the model makes something up and delivers it like a fact. There's a whole big conversation about trusting AI in general, but this post is about something narrower: when a model hands you an answer, how do you know it didn't just make it up?

When I was building my ResumeHelper app to help with my job search, I wanted to use AI to help reduce the toil. If my tool tells me "this role is perfect" when it isn't, that costs me a real day of my life where I'm tailoring a resume, writing a cover letter, and applying for a job I never had a shot at. That doesn't save me time, it wastes it! I didn't want to build something that wasted my time like that.

So I built a scorer-skeptic-judge system to fact check what was generated.

a job posting arrives rates the job against my real criteria SCORER tries to poke holes in the score rule: every objection needs a direct quote SKEPTIC weighs both sides and decides hard cap: one aggressive critique can't tank a good job JUDGE a score I can actually trust three models checking each other, not one talking to itself

I don't let one model decide

When the app scores a job, it doesn't just ask a model "is this a good fit?" and take its word for it. It makes the models argue about it:

Three models checking each other instead of one model talking to itself. I've found the disagreement is where the value is.

The cheap model does the cheap work

Not every question needs the smartest (and most expensive) model. So the app uses tiers: a fast cheap model for the obvious gut-checks, a mid-tier model for the real scoring, and the expensive one only for the hard stuff: writing, and the final judgment. There's also a fail-fast step: a cheap red-flag check runs first, and only jobs that survive it get the full scoring treatment. Most of the bad matches get filtered out before they cost anything.

Plain old code goes first

The app pulls salary ranges out of job descriptions. The easy version is "just ask a model." But a regular old regex finds a clean salary range instantly and for free, so that runs first. The model only gets involved for the genuinely tricky cases, like when a posting lists different ranges for different regions and something has to judge which one applies to me.

I test it like real software

My years in software development have taught me to rigorously test my code if I want to rely on it. This is the part that makes it a real tool instead of a demo. When I improved the scoring logic, I ran the new version against the old one on my actual saved jobs and compared them. One role I genuinely wanted jumped from 2.5 up to a 4 (out of 5) because the new version finally read all of my criteria instead of just a fraction of them. The pieces underneath have real tests too, and they have to pass before anything ships.

What I learned

You can't make the model trustworthy. What you can do is build a process that checks its work. Make it argue with itself. Make it cite its sources. Check it against reality. And when a regex will do the job, write the regex.

The robot helps me find a job. I still don't totally trust the robot, but it's helping.