From 10% to 97%: Seven Lessons From Taking an AI Workflow to Production

Australia has about 9,700 schools. My first AI scraper for School Scout took about fifty rounds of fixes. After all that, only 10% of those schools had data I'd show a parent.

Then I built a workflow engine, Graphstep, with evals in it from day one. Now 97% or more have data I'd show a parent. For the rest, I know why each one failed.

A better model helped. Better prompts helped. But the biggest change was simpler: a way to tell if a change made things better. Everything else in this post follows from that.

The short version:

1. Without evals, iteration is just motion.
2. Check 50 answers by hand. They become your golden set.
3. AI judges. Code checks the contract.
4. Grade every answer: golden, silver or revisit.
5. Pre-MVP, delete the unit tests. Keep the end-to-end tests.
6. Model IQ is a real variable. Measure it on your hard cases.
7. Use the least intelligence that does the job.

The problem

School Scout shows parents the fees, facilities and contacts of every school in Australia. Government data covers enrolments and results. It doesn't cover what a school charges, or whether it has a pool.

That information lives on each school's website. Sometimes it's on a fees page. Sometimes it's in a PDF two links deep, or in an enrolment booklet, or in a flipbook built in 2014 by someone who has since moved overseas. Every school uses a different layout.

Facilities can be approximate. Fees can't. If I say Year 7 costs $24,000 and it really costs $28,000, a parent makes a decision on a wrong number.

What Graphstep is

Graphstep is a workflow engine I built for this problem. It's private for now. Its first name was durable-step-functions, which tells you two things: what it does, and that I shouldn't name things.

• Workflows are files. YAML says which steps run, in what order, with which model and prompt. Small Go blocks do the logic.
• Runs are durable and live in SQLite on my machine.
• Every page fetch and model call is recorded, so I can replay any run.
• Agents write and change the workflows. I use Claude Code and Codex.

Low-code tools exist so that humans don't have to write code. But humans don't write most of my code any more, and an agent edits a file faster than I can drag a box. My first workflow put all the logic in YAML. It grew to 3,990 lines, which is less a config file and more a hostage situation. I moved the logic into Go, and the YAML went down to 1,340 lines with the same outputs.

When agents write the code, give them code.

Lesson 1: Without evals, iteration is just motion

Version one was simple. An AI agent wrote a TypeScript scraper with Playwright. It got some details from some schools, and it broke often.

So I iterated. With agents, iteration is cheap: describe the bug, get a fix, run it again. I did that about fifty times.

Every round felt like progress. It wasn't. When I fixed one school, another school quietly got worse. I only checked the schools I happened to look at that day, so I couldn't tell a gain from a regression.

My planning note from 9 September said it plainly: only 989 schools were parent-ready. That's 10%.

Fast iteration without a scoreboard is a random walk with good vibes.

So I started again, and Graphstep made every run an eval. The key is replay. I can change one step and replay a recorded run. Steps whose inputs didn't change come back from the record, in seconds and for free. Only the part I changed runs again.

That gives me the scientific method. Change one variable. Keep everything else the same. Compare the results.

When a school's answer changes, the harness goes through the AI steps in order. It names the first step whose answer changed, and it tells me why:

• Same request, different answer: the model drifted.
• Different input: the crawl read other pages.
• Different prompt: I changed something.

"Fees got worse" becomes "the fee step got a different PDF". That's a crawl problem, not a model problem, and I know where to look.

Lesson 2: Check answers by hand

An eval needs correct answers to compare against. So I checked 50 schools line by line against their websites. These are my golden set.

It isn't glamorous. Nobody at a party has ever leaned in when I said, "I read fifty school fee schedules."

But it improved results more than any prompt change. The golden set worked like a training set for the whole workflow. Each difference between a run and a golden answer showed me the next thing to fix: a page the crawl missed, a prompt that was unclear, or a rule that was wrong.

A review UI made the golden set much faster to build. Each workflow extension in Graphstep has its own review view, so whatever the task, I can quickly see what a golden result looks like. For School Scout, I click a school and see each fee and below its source quote, with any change from the golden run highlighted. If it's right, I click "Keep as golden".

AI helped me build most of the golden workflow: the review page, the comparison against each run, and the scoring. It took a few iterations to get to the version I have today. That version doesn't need a person to check new schools. The 50 golden answers keep the automatic checks honest.

When a person has checked a school, their answer always wins.

A small golden set that a person trusts beats a big one that nobody checked.

Lesson 3: AI judges. Code checks the contract.

This one is embarrassing.

My AI step made a call. Then my code decided which part of that call to keep. That can't end well. The AI step is there because the input is unstructured plain English, and IF statements can't cover it. If code could make the call, I wouldn't need the model.

I know how it started. One coding agent added a code check after an AI step. On its own, that made sense. Then every later session saw the pattern and copied it. That's what coding agents do: they follow the patterns that already work in your codebase. A bad pattern gets in and grows like a weed. The slopification of a codebase.

By early October, the fee section had 3,948 lines of these rules. They decided which charges count towards a year's total, which year a schedule applies to, how many terms a fee covers, and which campus a table belongs to.

Each rule probably made sense on its own. Together, they were the biggest source of wrong answers.

Here's the clearest example. One school printed a fee table with a "Total Fee" column. A separate table further down listed two compulsory levies that the total left out.

• The model read both tables, counted everything a family actually pays, and got $1,284 for Foundation. Correct.
• My code had a rule: compulsory fees must match the total that the school prints. It saw the "Total Fee" column, decided the model was wrong, and pulled the answer down to $786. Wrong.

The model read the school's website better than the school's own table did. My code took the table's side.

It wasn't a one-off. After every AI step, a code gate was waiting.

So I deleted the rules, and I added one line to the top of the project's instructions:

AI judges. Code checks the contract.

Now code only checks what a contract can check:

• Shape: the answer matches what the website imports.
• Required items: every campus that teaches a year has fees for that year.
• Grounding: every quote the model cites exists on the page it cites.
• Arithmetic: the numbers the model counted add up.

When a check fails, code never "fixes" the answer. It sends the answer back with the problem named, to a stronger model or with more pages to read. It does this two times at most. If the answer is still wrong, it goes to a person as a review row, the revisit register. Everything else ships.

The same commit took the School Scout profile code from 14,696 lines to 5,511.

After the golden set was done, most of my iterations looked the same. I edited a prompt, or I added or removed context in an AI step. The code checks didn't change: they only verify the schema and the shape. Then I replayed the run and compared the grades.

Lesson 4: Grade every answer: golden, silver or revisit

Graphstep doesn't keep a separate eval set. Every production run is the eval.

At the end of a run, each school gets one of three grades:

The site publishes the golden answer if one exists, else the silver one. A person never has to look at a school before it ships.

This contract is what made iteration fast. I don't need to check schools by hand to know if a change helped. I need to know how many schools moved between grades, and why.

Today, 97% or more of schools pass as golden or silver. The rest go to revisit.

But the failures matter more than the 97%. An accuracy number tells you how much went wrong. A failure list tells you what to do next. Here's one batch of 52 failures:

I expected most failures to be bad websites. About half were: 24 of 52. The other 28 were mine, and they clustered. That told me which prompt to work on next.

That split changes the conversation. "97% accurate" starts an argument about the 3%. "Half of the 3% is bad source data or sites we can't read, half is ours, and here's the fix for the biggest group" ends it.

An accuracy number starts an argument. A failure list with owners ends it.

Lesson 5: Pre-MVP, delete the unit tests

This is the one that starts arguments. If you're already typing a reply, please finish the section first.

By 3 October, I'd deleted every committed unit test in the repo: 13,704 lines over the project's history. I had two reasons.

Agents write lots of useless tests. Ask for a feature and you often get the feature plus 200 lines of unit tests that assert a function returns what it returns. They look like diligence. Mostly they're noise.

Tests make agents defensive. An agent that sees a wall of tests protects every behaviour those tests pin. It adds compatibility shims. It keeps old paths alive. In a pre-MVP repo that changes shape every week, that's backwards. I want the codebase to stay small, simple and clean, and the tests pulled the other way.

Also, the bugs that mattered weren't in the code. A prompt change made the model count a levy it used to skip. A school moved its fees PDF. A model update labelled things differently. No unit test sees any of that. The grades in Lesson 4 do.

What I kept: end-to-end checks on real runs, the way a person uses the system. Those are valuable at every stage. Agents still write throwaway tests in a session to check their own work. They delete them before the session ends.

I'll write unit tests again when the shape settles. Not before.

Pre-MVP, unit tests pin the behaviour you're still trying to change. Judge the system on real runs.

Lesson 6: Model IQ is a real variable

Early on, my fee reader was gpt-6-luna, with Meta's Muse Spark 1.3 as the fallback. I also tried Claude Haiku 4.5 and Muse Spark 1.2 in that role.

They're good models. But I was asking them to read fee schedules that a school committee wrote, put in a PDF, and then put in a flipbook. That isn't a fair test. It is the real one. With long, confusing context like that, each of them made wrong calls now and then.

Luna's mistakes weren't the normal "slightly different wording on each run" kind. Luna misread what the yearly fee was. At one boarding school, it read a note that said "Year 12 is charged over three terms" and applied it to Year 11 too. Year 11 tuition came out at $15,960 instead of $21,280. At another school, it returned 20 fee facts where earlier runs returned 70, and nothing sent it back.

I only caught it because of the fallbacks. When Luna hit errors and Muse Spark 1.3 took over a step, Muse got those cases right. It's like finding out your mate cheats at Monopoly because someone else had to play his turn.

When I moved up a size in each model family, the wrong calls went away: Haiku 4.5 to Sonnet 5.5, Luna to gpt-6.1-sol, and Muse Spark 1.2 to 1.3.

Then I ran a proper comparison. Three hard schools, the same pages to every model:

Two things stood out.

On clear extraction, strong models agree. Prices in a table are a solved problem at this level. Every dollar matched.

They split on judgement. What counts as a facility? Which sites count as campuses? Which charges does a family actually pay? Even one model gives different facility counts from run to run.

I chose Claude Sonnet 5.5 first in every role. It handled the messy structure best, and it's fast: across every run I've stored, it averaged about 14 seconds a request, against 24 to 45 seconds for the others. The other models stay as fallbacks.

Benchmarks tell you who's smartest on average. Your hard cases tell you who's smart enough for you.

Lesson 7: Use the least intelligence that does the job

Less intelligence is faster, cheaper and easier to repeat. Lesson 6 finds the minimum on your hard cases. This lesson is about using no more than that.

Not every question needs an LLM.

Most of a crawl is triage. Is this link likely a fees page? Does this window of text mention a pool? Does this page image show any dollar amounts? Those are yes/no and multiple-choice questions.

For those I use a System One model. The name comes from Kahneman's Thinking, Fast and Slow: System One is fast and intuitive, System Two is slow and deliberate. My System One model is Jev, from TypeSafe. Jev answers typed questions, and it returns a calibrated probability with each answer. That's what makes it good at ranking.

An LLM can answer the same questions through a JSON schema. Graphstep falls back to that when Jev is down. But the LLM's probabilities are all-or-nothing, and its confidence is self-reported. So it ranks badly.

Jev answers in about a quarter of a second, about 4,000 times per 100 schools. Early on, that swap made the whole workflow about three times faster.

TypeSafe, if you're reading this: please sell me a subscription!

The rule that made it work: rank, don't gate. The System One model puts the pages in order. It never removes a page from what the LLM can read. A System One model that gates will silently starve the LLM, and later you'll blame the LLM.

I learned that the hard way too. At first, Jev's link scores chose which fee links to follow. But the fees were often a few clicks away: a fees page, then an association's page, then the school's PDF. A link score can't see three clicks ahead.

So I built a bounded agent inside the workflow. The LLM reads the pages so far and picks up to four links to follow. The workflow fetches them and gives their links back to the LLM. This continues hop by hop, up to four hops, until the LLM has the fee documents or decides there's nothing left.

Of the pages the agent chose to follow, 46% were cited in the final answer. Of the fee pages I used to read up front, only 13% were. Fee-model input dropped by three quarters.

That's the agent loop: look, choose an action, see the result, choose again. The model chooses. The workflow owns the loop, and that gives me four things:

• Every hop is a durable step. It's retried, cached, budgeted and shown in the graph. A crashed run resumes from the last hop, not the start.
• Replay still works. I can rerun a recorded run against new code for $0.
• The eval still works. "Name the first AI step whose answer changed" (Lesson 1) needs the same steps in both runs. The hops are fixed slots, so it has them.
• The cost is bounded. Four hops of four links, set in YAML. An unbounded agent's bill is whatever it decides it is.

Anthropic's Building effective agents draws the same line. A workflow runs an LLM through code paths you wrote. An agent lets the LLM direct its own process. Start with the simplest thing that works, and add autonomy only where it pays.

Where would I use a full agent? Where the path is different every time:

• Schools whose fees aren't on a readable site. Some need a search on another site, a flipbook, a form, or an email to the office. Ideally, a person approves the last step.
• Schools whose register URL isn't theirs. Finding the real site is open-ended research.
• The revisit queue. Read the reasons, then re-run, change a prompt, or escalate.

An agent's answer goes through the same contract and the same grades. An agent is another way to produce an answer. It doesn't get a different eval.

I already use agents every day, at build time. Claude Code/Codex writes the workflows and the Go blocks. At build time, a mistake costs a replay. At run time, a mistake reaches a parent.

How I choose the kind of thinking for each step

The questions I ask of each step, in order:

1. Can code check it? Then code does it, and no model sees it.
2. Is the answer one of a fixed set? Use a System One model.
3. Is a wrong answer cheap or expensive? Cheap means it only changes the order, so the System One model ranks. Expensive means it hides something from a later step, so the LLM decides.
4. Does it need reading across pages, or a judgement about what counts? Use an LLM, and test it on your hard cases.
5. Can I write down the next step before I see this step's answer? If yes, it's a workflow step. If no, it's a job for a bounded agent.

Each wrong choice in this post broke one of these rules. Code did a model's job (Lesson 3). A model was too weak for its role (Lesson 6). A System One model did an LLM's job (the link scores, above).

Use a workflow when you can write the path down. Use a bounded agent when you can't. Grade both the same way.

The playbook: from AI pilot to production

None of this is specific to schools. Every team has the same problem when an AI pilot needs to become a production system. Here's the playbook I'd hand to a customer team, or to a partner's delivery team:

1. Agree on the golden set with the business owner. Take 10 to 20 real cases, more or less depending what it is. Have them checked by hand by the person who'll be blamed if the output is wrong. (Lesson 2)
2. Write the contract and the grades before the pilot starts. Define the shape, the required fields, and a quote in the source for every claim. Decide what ships with no person, what goes to review, and what counts as silver. The contract can evolve, and we should evolve it ASAP because migrating to the new contract really slows you down. (Lessons 3 and 4)
3. Let the model judge, and let code check the contract. When the model is wrong, change what it sees, how you prompt it, or which model it is. Change one thing at a time. Don't bolt a rule on after it. (Lesson 3)
4. Make every run replayable. Then each change is an experiment, not a guess. (Lesson 1)
5. Size the model on your hard cases, not on a benchmark. Model benchmarks are only a rough guide, you need to test on your own data on your own use cases. (Lesson 6)
6. Give each step the least intelligence that does the job. Code for checks, a System One model for closed questions, an LLM for judgement, and a bounded agent where the path changes every time. (Lesson 7)
7. Test end to end, at real volume. A demo set of a dozen cases hides the dead URLs and the blocked sites. Many edge cases only appear at scale. (Lesson 5)
8. Report failure types and owners. That report shows what's left, who fixes it, and whether you're ready for production. (Lesson 4)

You don't need a special platform for any of it. You need a golden set that a person trusts, a contract for everything else, and every run scored, case by case, against both.

If you're also working out how to get AI pilots into production, I'm always happy to compare notes.