The harness told me my AI was worse than I thought
I built a system to measure whether the assistant I'd shipped was actually any good. Then it told me it was worse than I thought — and that's the most useful thing that happened to the project all quarter.
Let me explain, because I think a lot of people are quietly stuck in the same place.
If you're building anything on top of an LLM — an internal assistant, an agent that reads your docs, anything that looks up your own material and answers questions from it (the usual name for that pattern is RAG — retrieval-augmented generation, which just means “look things up first, then answer”) — you eventually hit one uncomfortable question:
How do you actually know it's good?
Not “the demo worked.” Not “it felt right.” Know. With a number you'd stake a release on.
So I built the rig
What you build is called an eval harness — a test rig that takes a fixed set of questions, runs each one through the system, and has a judge score the answers along dimensions you care about: is it correct, is it complete, is it faithful to the sources it cites.
Think of it as the unit test of the LLM world, except the assertion is fuzzy and the thing doing the asserting is itself a language model. Hold that thought; it matters later.
Where it got humbling
While my test set was small, the headline correctness score looked great.
The moment I expanded it to cover the questions people actually ask — the harder ones, the edge cases, every category rather than the ones that were easy to write — the correctness score dropped. A lot.
My first instinct was the obvious one: the harness is broken.
That time, it wasn't. The small test set had been flattering me, in a direction I had every incentive not to question. The harness had flattering problems of its own, and they took me the next two months to find: questions with no answer key scored far more generously than questions with one, and the same run repeated with nothing changed flipped about a fifth of its verdicts.
A green dashboard you trust too early is more dangerous than a red one you understand. Small evaluation sets systematically flatter you — and until the set covers the full question surface, “improvement” is unfalsifiable.
That last word is the one that matters. If your eval can't come back and tell you that you got worse, then it can't tell you that you got better either. It's a mirror, not an instrument.
What the honest number bought
The deflated score changed a decision. I had wanted to hand the assistant to non-technical colleagues that day. Instead the rollout was held, with named reasons attached — and a specific, addressable gap to work on rather than a vague sense that things could be better.
That's the trade. You give up the comfortable number and you get the ability to make decisions you can defend.
Where I'm heading — and where I'd want input
I've started treating context like code: build a candidate version, gate it on measurement, promote it by hand, keep a known-good version to roll back to, watch it in production, repeat.
But the honest truth is that the measurement layer is the hard part, and I don't think anyone has it fully right yet. The questions I keep circling:
- How do you trust the judge scoring your system… when the judge is also an LLM, with its own biases you haven't characterised?
- What score is actually “good enough to ship” — and how did you arrive at that number instead of guessing it?
- How do you keep an eval set honest as the product grows, without it quietly drifting back toward the easy questions?
- Per-category thresholds, or one global bar?
If you've built evals for a RAG or agent system in production, I want to hear how you did it — what worked, and what you'd never do again.
Not looking for the magic. Looking for the measure.