Lab note · August 2026
Our benchmark lied to us. A 40-line script proved it.
Why we now verify the prompt instead of the counters. The results get their own post once the numbers are frozen.
In a day and a night, we built something we had wanted for months. Octave Engine is the multi-resolution memory compression layer of Mnemosyne OS. The same memory exists at several sizes. The engine picks the size when it injects the memory into a model's context.
Its rule fits in four words: Projection, never paraphrase. Every compressed form is made of verbatim extracts of the source, with their exact character offsets. A substring check proves that nothing was invented. A summarizer might round a date or improve a sentence. For a product that sells auditable memory, that summarizer is a liar with good intentions.
Over two thousand unit tests stayed green throughout. Then we put the engine on the bench. An A/B harness ran a control arm against a projection arm, on a long-term-memory QA corpus. An LLM judge scored the answers, and we logged everything. The deltas showed a shape. We wrote a rule into the design doc about when the engine should activate. We thought we had measured.
Then the founder asked one question, almost in passing:
"Verify the code. Not just the tests behind the tests."
The adversarial code review
We handed the entire diff to adversarial reviewers. Their brief: don't check that it works; try to prove it doesn't. The same mind wrote the code and the tests. So the tests had the same blind spots. Only fresh, hostile eyes find them.
One of their findings stopped everything.
Our harness fed the pipeline "frozen" memory rows, already retrieved. In the projection arm, we rewrote those rows in memory with the compressed text. But the prompt builder does not read in-memory rows. It re-reads each row by its hash, straight from the database. That is a reasonable cache pattern. So it silently discarded our modified copy and re-read the original.
Both arms had been sending byte-identical prompts the whole time.
Every delta we saw came from judge and model variance. Our careful rule rested on that noise. The noise had taken the shape of our hypothesis.
Why we never saw it
If you run LLM evaluations, this part will feel familiar.
The harness logs were full of real activity: entries projected, characters saved, fallback counters ticking. None of it was fabricated. The projection really compressed. But nothing reached the prompt. The instrument measured its own effort instead of its effect.
A busy log does not prove that the work reached the judge. Plausibility fooled us too. When the noise sketched a pattern, we could explain it, with support from the literature. A plausible explanation proves nothing.
Temperature zero does not protect you either. The answer model and the judge model are almost deterministic, and the two "almost" add up. On small samples, they produce gaps that look like findings.
The 40-line proof
The fix to the harness was small. The fix to our method matters more, and you can reuse it.
Before any paid evaluation run, a check script takes one memory row. It replaces the content with a marker string. It pushes the row through the exact path that builds the benchmark prompt. Then it checks both directions:
- With the fix, the marker must appear in the final prompt. So the arm really reaches the artifact.
- Through the old path, the marker must not appear. So the bug was the one we thought.
The script is about forty lines. It runs before the first judge token is spent. If it fails, the campaign does not start.
Our rule: an A/B experiment exists only once you have proven the arms differ in the final artifact. The config, the logs and the counters are not enough. What counts is what the model sees.
What we did with the invalidated campaigns
The invalidated campaigns stay in the design doc. They are struck through, tables intact, next to the corrected ones. The negative results stay too, each with the cause of its failure. A research log that keeps only the successes turns into a brochure.
Then we re-ran everything on the proven instrument. Those numbers are real. They will be published separately, frozen, with their sample sizes and caveats.
An engine designed by three voices
Octave came out of a three-way conversation. A human had the initial intuition and the final say. An AI pair-engineer worked in the codebase. The assistant built into Mnemosyne OS reviewed the designs against its memory of the project.
A summary of the work session went into the OS's memory. Minutes later, the assistant used it to propose the next experiments. When the numbers were invalidated, part of that memory was invalidated too. The correction is a new chronicle that supersedes the old one without erasing it.
This memory can be audited and corrected in the open. We build the product the way we work. This week, our way of working got tested first.
Mnemosyne OS is a desktop app for Windows, macOS and Linux. Download Mnemosyne OS