Building your own tools · Level 4

Eval loops

20 min intermediate

You handed a job to an agent on Friday afternoon and went home. Monday morning there are eleven files waiting, and nobody has read a word of them.

So you read all eleven, which is the work you delegated in the first place. Or you trust them and ship. An eval loop is the third option.

The eval loop

If you can judge whether a result is good, you can hand that judgment straight back to the agent and let it fix its own work. That is the whole idea. Everything else in this course is detail.

The judge can be any of three things, and none of them has to be you.

A script. It runs, and it either passes or it does not. The build compiles, the field exists, the number is under the limit.
A score. Anything you can put a number on. Word count, runtime, how many checks came back green.
Another agent. For everything a script cannot see: whether the argument holds up, whether the writing is any good.

Wire one of those to the output and the loop runs itself.

   1. Produce      the agent does the work

   2. Check        a script, a score, or another agent grades it

   3. Passed?      yes  ->  done, hand it back
                   no   ->  step 4

   4. Feed back    tell it what failed, then go to step 1

Step four is the one that matters. An agent that is only told “try again” guesses at what was wrong. An agent that is told what failed fixes that thing. The check is not there to catch the agent out, it is there to give it something to work with.

Produce, measure, keep or discard, a hundred times over while Karpathy slept. It beat him.

The judge does not even have to be the same model. There is a Codex plugin for Claude Code that hands work to a different model entirely, and people reach for it when they are stuck or want a second opinion on a diagnosis. A model trained differently has different strengths and notices different things, so it makes a good critic of work the first one was happy with. Two competent models rarely fail in the same place.

Where the loop breaks

The check is the loop’s entire definition of success, and whatever you actually wanted stayed in your head. That gap is where every failure comes from.

Karpathy’s loop cheated. Watchers caught it tweaking a setting that flattered the score without improving anything. It was told to move a number, so it moved the number the lazy way.

It goes wrong in two places.

A vague target. The check has to return yes or no. “Make it good” gives it nothing to decide on, so the loop fiddles forever. “Has a numbers section, every figure traces to the source” does.
A gameable target. The loop chases whatever you measure. Give it a number and it will move that number by the cheapest route it can find, cheating included. If your test passes when the agent deletes the test, the agent may delete the test.

Most of the work in a good eval loop goes into designing a target that cannot be faked. If the agent can touch the check, it can change the check. Point the check at something the agent does not control. Four moves do most of it.

Check a different surface. Do not read the agent’s claim that the tests pass. Run them.
Score a side effect. “I updated the file” is a sentence the agent wrote. “The file contains X” is a fact.
Use ground truth it cannot edit. Grade against a fixture, or a locked expected output.
Keep the check out of reach. If the agent can edit or switch off the check, the check proves nothing.

None of these needs a model. They need you to decide, once, what done means and where you would go to confirm it.

Heaven

Your target is tied to the real outcome and sits where the agent cannot edit it. The loop works at it all night, and every gain it reports is a gain you keep.

Hell

Your target is a single loose number. The agent finds the shortcut and moves the number without doing the work. You wake up to a green dashboard and no progress.

Checks and evaluations

Run the cheap checks automatically on everything, and save the expensive evaluation for work that has already survived them.

A check is mechanical. It runs a script and gets a yes or a no. The build compiles, the field is filled in, the number came back as a number and not as the string “N/A”. A check finds all of that in under a second, so there is no reason to ever run it by hand.

An evaluation is anything that has to exercise the work to find out. Sometimes that is a model reading the finished thing and telling you the argument is thin, the second slide repeats the first, the tone is wrong. Just as often it is not a judgment at all. Run it under load and time it. Put it through a simulation and see what comes out. Let it play the game and take the score at the end. Anything where you only learn the answer by putting the work through something is an evaluation, whether a model or a stopwatch produces the number.

The order is the whole trick. There is no point timing a build that does not compile, running a simulation on a config with an empty field, or asking a person what they think of work that fails its own tests. Every problem you catch cheaply is a problem you do not pay to discover expensively.

Once you see it that way, you can put more rungs on the ladder. A script, then an agent reading the output, then a real run or a simulation that produces a number, then a person looking at whatever survived all of that. Each step costs more than the last and tells you something the last one could not.

You are aiming to get the most useful signal into the loop as early as you can, at a cost you are willing to pay. Where that line sits depends on the job, and you will move it as you go.

A working carousel-building skill is laid out this way. A validator and a browser QA pass run on every single build. The three critics come after, once there is something worth reading closely.

When the evaluation is a model reading the work, two things make it worth what it costs.

Starve it of context on purpose. A real slide-critic agent gets one line in its instructions: you have NO access to the source JSON or generator code. A critic that can see the generator starts explaining why the output came out the way it did. It should see the artifact the way a reader sees it.

Brief it with an attitude. A real insight-critic opens with “You challenge the story. Assume the first draft is too convenient.” Every item comes back with one verdict: keep, revise, or cut. A stance and three words to choose between beat a list of boxes.

Heaven

Your critic sees the finished page and nothing else. It reads it cold, the way a stranger would, and tells you slide four says nothing.

Hell

Your critic sees the generator too. It reads the code, understands why slide four came out empty, and files a note explaining the design.

Getting a loop running with `/goal`

You do not have to build any of this. Claude Code ships a loop, and your job is to state the goal well enough that it can tell when it is done.

/goal keeps the agent working until a condition you state is met. A separate model reads each turn and answers yes or no, so the agent never grades its own work. A no starts another turn.

That judge only sees the transcript, so the condition has to be provable from what the agent produced. “The tests pass, shown by the run output” works. “The code is clean” does not. This is the same problem as a vague target, and it is where all your effort should go. Almost every disappointing /goal run is a goal that could not be judged, not an agent that gave up.

So write the condition the way you would write it for someone who will only ever see the output and never talk to you. Name the thing that must be true, and name where you would look to confirm it.

That covers a lot of ground. When you outgrow it, the next step is the full autoresearch shape: a file recording what has been tried, the experiments that failed and why, a score carried across runs so the loop knows whether it is getting anywhere. That is worth building for a problem you will grind on for days. For most work /goal is genuinely enough, and you can start using it this afternoon.

The outer loop

Everything so far enforces the standard you already wrote down. Nothing in it tells you the standard was any good.

That is the last rung on the ladder, and it is you. What people actually thought, what actually got used, what you found yourself fixing by hand anyway. Feed that back and the target itself gets better, not just the work that passes it.

Most people build the first loop and stop. Teach it your taste, next in this track, builds this one end to end.

Hands-on

First, watch a target get gamed. Pick something you want short and sharp: a bio, an announcement, a product blurb. Write it down, and list the three or four points it must keep. That list stays with you.

Hand Claude Code the loop, and give it one thing to chase:

Here is a paragraph. Run this loop on your own, ten rounds: each round, rewrite the paragraph fresh
with a genuinely different phrasing and count the words. Keep a version only when it beats your
current best on word count. After ten rounds, show me the winner, its word count, and what you tried
each round.

An editable thing, a score to push down, a fixed budget. The score is word count, and your must-keep list is nowhere in that prompt.

Read the winner against your list. If the shortest version won by dropping one of your points, you just watched the thing this course is about happen on your desk. Nothing cheated. The score rewarded shorter.

Now fix the target, not the agent:

Same paragraph, same loop, ten rounds, but the score changes. A rewrite only counts as a candidate if
it still contains every one of these points: [list them]. Throw out any rewrite that drops one, even
if it is shorter. Among the versions that keep all of them, the shortest wins.

Rewrites that would have won get thrown out before they are scored. Same agent, same budget, different target.

Now use the built-in loop. Give Claude Code a real job with something checkable in it, like a one-page HTML overview of your team, and decide what must be true when it is finished: the team name is on it, there is a “Contact” line, it names the current quarter.

Start the loop, with the condition written so the judge can confirm it from the output:

/goal The page contains the team name, a line starting with "Contact", and the current quarter.

When the agent says it is done and the judge disagrees, it goes back around without you saying anything.

Break it on purpose. Run the same job with a goal nobody could judge:

/goal The page looks professional.

It fiddles, or it stops on something you would not have accepted. Nothing failed. You gave a judge a question it had no way to answer.

Fix that one. Write down what “professional” actually meant to you when you typed it, as things a stranger could confirm by looking at the page, and run it again with that.

Reflect

Pick a task you hand to an agent while you are not watching. Name the check that would catch it failing. If it is “I read the output”, you are the loop.
Now try to game that check. What is the cheapest way to pass it without doing the work? Find a shortcut in a minute and so will the agent.