Lab note · August 2026
Our benchmark lied to us. Here's the 40-line proof that caught it.
Instruments, noise, and why we now verify the prompt instead of the counters. Results held back; they get their own post once the numbers are frozen.
We spent a day and a night building something we had wanted for months: Octave Engine, the multi-resolution memory compression layer of Mnemosyne OS. The idea fits in a sentence: the same memory, available at several sizes, chosen at the moment it is injected into a model's context. Its founding rule fits in four words. Projection, never paraphrase. Every compressed form Octave serves is made of verbatim extracts of the source, each carrying its exact character offsets. You can prove by substring, mechanically, that nothing was invented. For a product that sells auditable memory, a summarizer that might round a date or improve a sentence is not a compression engine. It is a liar with good intentions.
So we built it and we tested it (over two thousand unit tests stayed green throughout), and then we did what you are supposed to do: we put it on the bench. An A/B harness over a long-term-memory QA corpus, control arm against projection arm, an LLM judge scoring answers, everything logged. The runs came back. The deltas came back. We started to see a shape in them. We wrote a rule about when the engine should activate straight into the design doc, with the confidence of people who had measured.
Then the founder asked one question, almost in passing:
"Verify the code. Not just the tests behind the tests."
The review that earned its keep
We handed the entire diff to adversarial reviewers with a single mandate: don't check that it works; try to prove it doesn't. The tests were written by the same mind that wrote the code, so the tests share the code's blind spots. Fresh, hostile eyes are the only known cure.
One of the findings ended the party.
Our benchmark harness fed the pipeline "frozen" memory rows: pre-retrieved, and in the projection arm, rewritten in memory to carry the compressed body. What we had missed: deep inside the pipeline, the prompt builder does not trust in-memory rows. It resolves each row's content by its hash, straight from the database, which is a perfectly reasonable cache pattern. In doing so it silently discarded our mutated copy and re-read the original.
Both arms had been emitting byte-identical prompts the whole time.
Every delta we had seen (the ones that worried us, the ones that thrilled us, the rule we had so carefully derived) was judge-and-model variance wearing the costume of our hypothesis.
Why we never saw it
This is the part that should worry anyone running LLM evaluations, because it will feel familiar.
The harness logs were full of real activity. Entries projected. Characters saved. Fallback counters ticking for exactly the right reasons. None of it was fabricated. The projection genuinely ran, genuinely compressed, genuinely produced all those numbers. It just never reached the prompt. The instrument was faithfully measuring its own effort instead of its effect.
That is the failure mode. A log that looks like an arm doing work is not evidence that the work arrived in the artifact you are scoring. And there was a second accomplice: plausibility. When the noise happened to sketch a pattern, we could explain it. The explanation was mechanistically reasonable. It matched intuitions from the literature. It was satisfying. Plausibility is not evidence. It is what noise looks like when you want it to mean something.
Temperature zero does not save you either. An answer model plus a judge model is a stack of almost-deterministic components, and "almost" compounds. On modest sample sizes, that stack will hand you differences large enough to feel like findings.
The 40-line proof
The fix to the harness was small. The fix to our method matters more, and it is small enough to steal.
Before any paid evaluation run, a check script now takes one memory row, replaces its content with a marker string, and pushes it through the exact prompt-assembly path the benchmark uses. Then it asserts both directions:
- with the fix in place, the marker must appear in the final prompt, proving the arm actually reaches the artifact;
- through the old path, the marker must not appear, proving the bug was what we thought it was and not something stranger.
Forty-odd lines. It runs before a single judge token is spent. If it fails, the campaign does not start.
The rule that came out of it: an A/B experiment exists only once you have proven the arms differ in the final artifact. Not in the config. Not in the logs. Not in the counters of your instrument. In the thing the model actually sees.
What we did with the corpses
We did not delete the invalidated campaigns. They sit in the design doc, struck through, tables intact, next to the corrected ones, because a research log that only contains survivors is not a log, it is a brochure. The negative results stayed too, each with its failure mechanism spelled out. A team that shows you what it got wrong tells you how much to trust what it got right.
Then we re-ran everything on the proven instrument. Those numbers are real, they are interesting, and they are not in this post. They will be published separately, frozen, with their sample sizes and their caveats attached, once they have survived the scrutiny they deserve. The day before the numbers is where evaluation work is won or lost.
Three voices, one memory
One more thing about how this engine was built. Octave was designed in a running three-way conversation: a human with the initial intuition and the final say, an AI pair-engineer in the codebase, and the assistant that lives inside Mnemosyne OS, reviewing designs against its own memory of the project. At one point a summary of the work session was ingested into the OS's memory, and minutes later the in-app assistant was reasoning from that very chronicle to propose the next experiments. When the benchmark numbers were invalidated, so was a piece of that ingested memory, and the correction was written back as its own chronicle, superseding without erasing.
Memory that can be audited, corrected in the open, and never silently rewritten. We build the product that way because we work that way. This week our way of working got tested first.