One retrieval rule fixed enumeration and broke explanations
“What are all the X?” came back with three. The real answer was forty. The fix worked perfectly — and quietly degraded conceptual answers.
Two things you need first, and they take one sentence each.
Long documents get sliced into smaller passages before they're stored, because a whole document is too big to hand a model — those passages are called chunks. And when a question comes in, the system doesn't read everything; it grabs a fixed number of the closest-matching chunks, say the best handful. That's it. That's the machinery.
Now: a user asks for a complete list of something. The system fetches its usual handful of best-matching passages. The model, working from those, confidently produces a list.
The list is wrong, and nothing anywhere in the system knows it's wrong.
This is a structural problem, not a model problem. Search is built to surface the most relevant passages, not all of them. If the forty things you asked about are scattered across eighty documents, fetching the best handful hands the model a sample — and the model does what anyone does with a sample they don't know is a sample: treats it as the whole population.
The fix that worked
The answer was to stop pretending retrieval could discover an exhaustive set, and instead build one:
- Build the list once, from the source of truth. Don't hope search discovers all forty items — go read them straight out of the code or config where they're actually defined, and write them into a single document.
- Store that document whole — never sliced. This is the crux. If the slicing step cuts a forty-item list in half, retrieving one half gives you a confident answer that's missing twenty things. A list is only worth anything intact.
- Put it first in the results when the question is asking for everything.
Enumeration went from three items to forty verbatim identifiers. Clean win. Ship it.
The part I didn't see coming
The list leaked.
The list document was large, dense and never sliced, so almost any keyword matched something inside it. It ranked highly on questions that weren't asking for a list at all — “how do I debug…” — where the person wanted an explanation, not an inventory.
The answers didn't fail loudly. They came back as walls of constants instead of explanations — technically grounded and much less useful.
The queries it degraded were disproportionately the conceptual, non-technical ones — exactly the questions a new or non-engineering user asks first. The fix optimised for the power user at the newcomer's expense, which is the opposite of what you want from an adoption standpoint — and it is why the rollout to non-technical users was held.
The actual fix: stop having one ranker
The mistake wasn't the list. It was assuming one set of rules for ordering results could serve every kind of question. One ranking rule, every question shape, no exceptions — that was the actual bug, and it was a design decision nobody had ever written down as one.
The fix I designed starts by asking a different question first: what kind of question is this? A request for everything, a request for an explanation, or a lookup of one specific thing — each with its own rules for what to put in front of the model, so the big list goes first for “show me all of them” and gets pushed down for everything else. I haven't shipped it for the list. The same pattern did go in for another kind of document, which now only enters the results when the question is actually about it — and that gave back the facts it had been crowding out.
The technical name is intent-conditioned retrieval. The plain version is better: decide what's being asked before you decide what to go and fetch.
Per-query-shape ranking, not a single global ranker. The right retrieval strategy is a function of what's being asked, and treating it as a constant is a bug you'll only find by looking at the queries you didn't optimise for.
Three things I took from it
- A win on one query class is a hypothesis, not a result. I measured enumeration improving and shipped. I hadn't measured what happened to everything else, so I didn't know — and “didn't know” looked identical to “fine.”
- Category-balanced evaluation isn't bureaucracy. A global average can easily hide a real regression in a category that matters more for adoption than the one you fixed.
- Retrieval failures are silent by construction. Code that breaks throws. A retriever that returns the wrong passages returns an answer — fluent, confident, sourced, wrong. There's no stack trace for “you retrieved a plausible thing instead of the right thing.”
Which is the whole argument for measuring per-category rather than globally. The global number is almost always fine. That's the problem with it.