I wrote the counter-arguments before I wrote any code

There's a version of my tool that's faster, lighter and more elegant than the one I shipped. In it, I've fine-tuned a small model on the GOV.UK Design System until the house style lives in its weights. The system prompt shrinks because the model already knows the 473 classes. A fine-tuned 2-billion-parameter model matches a stock 4-billion one, so I drop down a size tier and everything runs quicker on a modest laptop. It's a lovely story.

It might also be a trap. So before I wrote a line of training code, I wrote the decision memo that tries to talk me out of it. This post is that discipline, because I think "write the counter-case first" is the most useful PM habit I have, and an ML feature is a perfect place to show it.

The case is genuinely strong — that's what makes it dangerous

I want to be fair to the idea, because weak ideas are easy to kill and strong ones are the ones that waste your quarter.

Fine-tuning is usually a gamble because you can't cheaply tell whether it worked. This domain is the rare exception, for four reasons. The output is narrow and closed: valid markup is restricted to 473 known govuk-* classes with strict structural rules. A mechanical reward signal already exists: my deterministic scorer grades any page 0–100 with no human in the loop. There's a reusable corpus: 161 scraped Design System examples, 37 component docs, and the gold eval cases to seed synthetic training data. And because the scorer is mechanical, improvement is objectively measurable rather than a matter of taste.

That really is close to the textbook case for fine-tuning a small model. The prize is real: a shorter prompt and a smaller, faster model clearing the same quality bar on the weakest device I support — the government standard laptop, an Intel i5 with 16GB and no GPU.

When an idea is this attractive, the responsible move isn't to start building. It's to find out, as cheaply as possible, whether the attractive version survives contact with reality.

The counter-case, in writing, before committing

Here are the load-bearing objections I made myself answer first. Not vague risks — specific ways the project fails.

The whole economic case rests on one unproven claim. "A fine-tuned 2B beats a stock 4B" is the entire justification. If it's false — or true at full precision on a training GPU but false once I quantise to 4-bit and run it on an i5 — there is no payoff at all. Everything depends on a claim I hadn't tested.

My evaluation is seven cases, and that's not an evaluation. Seven cases is a smoke test. You cannot make a ship decision, or detect overfitting, on seven examples. Expanding the held-out set to a hundred-plus cases isn't a nice-to-have; it's the prerequisite that makes every other number trustworthy. Until it exists, the "objective measurement" advantage I was leaning on is partly illusory.

Quantisation can quietly eat the win. I ship a 4-bit model. A quality gain you measure at full precision on a powerful GPU can shrink or vanish at 4-bit on a weak CPU. So any gain has to be measured on the deployed quantisation, on the actual target device — never on the comfortable training rig.

A fine-tune freezes the house style; the Design System keeps moving. My retrieval approach refreshes by re-running a script when the Design System ships a new version. A fine-tuned model bakes the style into its weights and has to be retrained to keep up. I'd be trading a cheap maintenance task for a recurring training project, indefinitely.

Licensing is a hard gate for a government-adjacent tool. Redistributing fine-tuned weights in an installer means clearing the base model's terms, Crown copyright and the Open Government Licence on the GOV.UK-derived training content, and the Design System's own licence. That's a question to answer before investing, not after.

There may be a lazier win that's almost as good. Maybe just shipping the smaller model as the default, trimming the prompt and pruning retrieval already buys most of the speed — for none of the training cost. If "do almost nothing" gets me 90% of the way, the fine-tune has to justify itself against that, not against the original slow baseline.

Turn the objections into gates

A counter-case is only useful if it changes what you do. So each objection became a go/no-go gate, ordered cheapest-to-falsify first.

Gate A — before any training. Does a hundred-plus-case eval set exist, and does the "lazy alternative" control not already hit my speed and quality targets? If doing almost nothing already wins, I ship that and shelve the fine-tune. If I can't grow the eval set, I stop — I'd have no way to trust the result anyway.

Gate B — after a one-day spike. On a small dataset, does the fine-tuned small model show a real, post-quantisation uplift over the stock small model? If not, the size-tier thesis is probably false, and I pivot or stop — having spent a day, not a month.

Gate C — before shipping. Are the quality and speed targets met on the actual laptop, is licensing cleared, and have I budgeted the retraining cadence? If not, the release holds.

The recommendation I landed on wasn't "do it" or "don't". It was: fund Phase 0 only — the eval set and the baselines — and treat the full pipeline as unfunded until Gate A and a Gate B spike are green. The expensive work stays gated behind the cheap proof.

Why this is a government habit, not a contrarian one

None of this is pessimism, and it isn't clever-for-its-own-sake. It's the same instinct government encodes in discovery and alpha: do the cheap, falsifiable work first, and don't build the expensive thing until the risky assumption underneath it has survived a test. "Start with the user need." "Do the hard work to make it simple." Writing the counter-case first is how you avoid the most expensive mistake in product work — spending a month proving something you could have disproven in a day.

I might still fine-tune the model. The case for it is honestly strong. But if I do, it'll be because it cleared the gates I set to stop myself — not because the idea was pretty and I was in a hurry.

Prompt to Page is an independent project by Courtney Allen. It is not affiliated with, endorsed by, or connected to the Government Digital Service, the Crown, or any UK government body. "GOV.UK" and the GOV.UK Design System are referenced descriptively.