What we measured, and what turned out to be wrong.

Two documents are archived with a DOI, released under CC BY 4.0 and attached to an ORCID. You can cite them. The DOI will still work in ten years, whatever happens to this site.

Technical whitepaper, DOI 10.5281/zenodo.21728283Benchmark audit kit, DOI 10.5281/zenodo.21727140ORCID 0009-0009-1087-3917

77.1% on LongMemEval-M, full-haystack variant, on 48 questions the engine had never seen, with the official LongMemEval judge. In this variant, each question is buried in about 480 distractor sessions. Most published numbers come from a smaller version.

The protocol was published before the run. Each question was answered twice, and it counts only when both answers are right. On the 48 questions used to tune the engine, the strict judge gives 85.4%. In September an outside audit found a wrong verdict in our August result: an erratum corrected it to 72.9%, and we ran the whole benchmark again. The reader, the engine and the judges changed between these runs, so the figures do not form a progression.

These figures can only be compared with results on the same benchmark, with the same judge.

Audit the score yourselfThe full dossier

We pre-register the protocols before the decisive tests. So nobody can adjust the hypothesis after seeing the result. Every arm is compared against control arms, some of them deliberately absurd. If a random control reproduces an effect, that effect is discarded.

Statistical comparisons use a permutation test with a fixed seed. When a language model grades an answer, we first measure its noise and publish it. A gap smaller than that noise is reported as noise.

For months, the founder thought that a particular mathematical structure would improve a memory engine. It was his favourite idea. We tested it on several independent tasks, in every possible form, including the one he considered right.

It brought no signal. Random controls reproduced each of its apparent effects. A structural analysis of the data found nothing to support it. We abandoned the hypothesis.

By running this test to the end, we found the design that did ship: consolidation that adds to memory instead of replacing it. It keeps the raw record and adds on top, without summarising.

The references are in the whitepaper, section 12: the “operating system” idea for LLMs, the benchmark we use, the closest published architectures, and the work on reproducible evaluations.

Whitepaper, §12 Related Work