What I Found Under My Own Chatbot

What I Found Under My Own Chatbot

What happened when a bodybuilding coach refused to hand-wave retrieval.

I coach bodybuilders. I did not set out to do AI research. This is the story of how the first quietly turned into the second — and why it matters for anyone building a chatbot on top of their own knowledge.

The complaint that started it

When I launched the AI assistants on TroponinIQ, I got the complaint every knowledge-base chatbot gets: "these feel generic."

That word stung, because the whole point was the opposite. The assistants are built on my coaching knowledge base — years of contest prep, nutrition, hormones, peak week, the actual methodology I use with athletes. If they sounded like a generic model, something between my knowledge and the answer was leaking the personality out.

The easy move is to blame the prompt and tweak wording. I did some of that. But the real problem lives one layer down, in the part almost everyone hand-waves: retrieval ranking.

The part everyone hand-waves

Here's how a system like this answers you. It pulls a handful of candidate passages from the knowledge base, and a component called a reranker scores and orders them. The top few become the evidence the model actually writes from. If that ranking is basically a coin-flip among a dozen relevant pages, your answer is generic by construction — the model is writing from whatever happened to float up, not from the best thing you wrote.

So the quality of the whole product depends on a scoring function most people never open up. It looks like this:

R = base + Σ (wₖ · fₖ)

Every fₖ is a signal — is this a worked example? does it share a link with the question? is it a canonical protocol page? — and every wₖ is a weight you get to tune. The entire game is choosing those weights well.

The thing nobody tells you about tuning

I started tuning the way everyone does: change a weight, run an evaluation, see if the numbers move. And the numbers wobbled in ways that didn't make sense. A change that should have mattered did nothing; a change that shouldn't have mattered moved things around.

Then it clicked, and it's a little embarrassing how basic it is. That scoring function is a linear map, and linear maps have kernels — directions you can move along and observe nothing. If two of your ranking signals are effectively redundant, then there's a combination of their weights you can crank up and down all day and the ranking will never change. Sweeping those weights isn't optimization. It's measuring noise. And nothing in a normal tuning workflow tells you which combinations are dead.

That's the whole trap: you can run a careful, expensive tuning sweep and have part of it be meaningless, and never know.

So I stopped guessing and did the linear algebra

Instead of sweeping weights and hoping, I built a diagnostic that computes those blind directions directly — from live production traffic, read-only, before a single tuning run gets wasted. It looks at how the signals actually co-vary on real questions, finds the combinations that don't change any ranking, flags the pairs of signals that are secretly redundant, and hands back a pin list: the exact weights to hold fixed so a sweep measures what it thinks it's measuring.

Then I pointed it at my own system, and the most valuable thing it did was tell me I was wrong.

• One pair of my signals looked entangled. Before I pinned them, I ran the control the tool's own logic demands — and the entanglement dissolved. It was an artifact of how I'd built the test pool, not a real property of the system. The tool caught its own false positive.
• With that cleared up, the production map came back well-conditioned and clean — no unstable directions, nothing that needed pinning.
• It also re-examined a weight sweep I'd already run and certified it as trustworthy — across roughly 2,500 feature vectors and 142 real query groups.
• All of it at zero cost to the live system: byte-identical to the production scoring path, so no added latency, and mutation-proven — break the underlying math and a test turns red.

That first bullet is the one I'm proudest of. Anyone can build a tool that finds problems. A tool that argues you out of a wrong conclusion — with a control you can rerun yourself — is the difference between rigor and a dashboard that tells you what you want to hear.

The physics detour (and an honest disclaimer)

The lens that got me here came from thinking about the scoring function the way you'd think about a physical system — through symmetry and conservation. In physics there's a deep result (Noether's theorem) that ties symmetries to conserved quantities, and the "directions that don't change the ranking" are exactly that flavor of idea: a symmetry of the system.

Now the disclaimer, up front and in bold, because I want the real researchers who read this to keep reading: this is Noether-type thinking, not a claim that I proved anything about Noether's 1918 theorem. Gradient descent isn't a nice Lagrangian system; the honest version is an elementary, checkable argument, not a grand unification. The analogy earned its keep because it produced a working tool — but I'm not going to dress it up as more than that.

Two write-ups are on their way to arXiv and machine-learning / information-retrieval workshops: a theory paper on designing symmetries into sparse networks and tracking their conserved quantities under training, and an experience paper on the production tool — "Identifiability Audits for Additive Rerankers." I'll link them here when they're up.

Why this makes the chatbot the opposite of generic

Here's the payoff, in plain terms. The AI in your corner on TroponinIQ is grounded in one coach's actual knowledge base, and the machine that decides which piece of that knowledge reaches you is tuned against measured identifiability instead of vibes — instrumented with a diagnostic that most production AI systems simply don't have. "Generic" is a system running on defaults nobody looked at. This is the other thing.

And there's a general lesson in here for anyone building retrieval on their own content: your ranking is probably part noise, and you can't see it by eyeballing a correlation matrix. The redundancy only shows up when you look at how signals move within each question — which is the non-obvious step, and the one worth stealing.

The ask

I built almost all of this alone, talking to nearly no one in this field. I'd like to change that. If you work on retrieval, ranking, or identifiability and I've reinvented a wheel — tell me. If it's useful — tell me that too, and tell me where to push it next. I'll take the correction over the compliment.

— Justin Harris, TroponinIQ