Docs · Method
Methodology
How a number earns its place on this site, and what has to be true before any of it is published.
Retrieval is not answer quality
Finding the right evidence and answering correctly are different things. We report them in separate columns, never added together.
The publication protocol
A number only goes public when all four hold:
- Pinned harnessProduced by the pinned public harness, never a private script.
- Full bundleThe complete artifact bundle is released at the same time as the number.
- Separate columnsRetrieval and judged accuracy in different columns, judge model and prompt disclosed.
- One commandA single documented command regenerates the number from the pinned bundle.
The private regression suite is never conflated with public numbers and never headlined. Third-party reproduction strengthens a claim when offered; it does not gate publication.
Guarding against gaming
- No tuning on held-out or test splits, Mnemosyne included.
- No self-defined benchmark is ever a headline claim.
- Experiments are pre-registered before scoring; off-registration scores are labelled ineligible.
- Every attempt, including failures, is signed into the append-only ledger.
- No top-k equal to the whole candidate pool, and no recall presented as accuracy.
Building the site is not publishing
This preview can show numbers that are not approved. A real release has to clear the gates on Operator and governance.