Methods
An evaluation is only reproducible if the instrument is available. We publish the specification we report against, the harnesses we build, the protocols we adjudicate to, and the rules that bound what any of it can establish.
Released methods
Harnesses are released with the first finding that uses them.
Harnesses and reference datasets are released with the first finding that uses them, so that the instrument and the measurement it produced arrive together.
What we can claim depends on what we were given.
Access is determined and recorded before method design, because it bounds the available conclusions more tightly than any other factor. The tier is stated on the front of every report and in the index entry for every finding. Claiming beyond the tier is the most damaging error available to us.
| Claim type | Black box | Grey box | White box |
|---|---|---|---|
| What we have | Queries and outputs. | Plus documentation, confidence scores, a description of the training data, and logs. | Plus weights and internals. |
| We may claim | That the system fails, where, and how often. | Plausible mechanisms, and calibration analysis. | Mechanistic explanation. |
| We may not claim | Why it fails. | Causal attribution to specific design choices. | That the system we were given is the system in production. Provenance is established separately, and usually cannot be established by us. |
No tier lets us certify that a system is safe. An evaluation establishes how one version behaved on one test set at one time. Generalising that to the deployed population is an inference the reader makes, and every report sets out what would have to be true for it to hold.
The protocol is published before the study is run.
Before execution begins we publish the method file, timestamped: the unit of analysis, primary and secondary metrics, the harm threshold and its justification, ground truth source and its known limitations, disaggregation dimensions, sample construction, statistical approach, and the uncertainty budget. It states the hypotheses and states what result would falsify them.
This costs nothing and makes post-hoc rationalisation structurally impossible. A reader can check what we said we would measure against what we reported measuring.
We establish what our tests detect before we trust them.
For each method we document what it actually detects, its sensitivity to specification choices such as prompt phrasing, detection thresholds and sample composition, its known false positive and false negative behaviour, and the range of systems over which it is valid.
Where a method is new, it is validated against a system with known ground truth before being applied to a study subject. This is the requirement most AI evaluation skips, and it is the largest quality gap between a laboratory and everyone else.
A validated method is valid over a stated range of systems and conditions, not universally. Every method file names that range. Applying a method outside it produces a number, and the number means nothing.
Released by default.
Harnesses, scoring code, and adjudication protocols are released by default. A harness that cannot be inspected cannot be checked, and a finding produced by an uninspectable instrument is a claim rather than a measurement.
Every published chart ships with the machine-readable results file behind it, so a reader can replot or reanalyse without asking us for anything. The notebook ships with the finding. Reference datasets we collect ourselves are released with the finding that first uses them, together with the collection protocol, the sampling frame, and the annotation guidelines.
If anyone reproduces our work and gets a different answer, we publish that too.
Some material cannot be released. Datasets containing personal data, imagery licensed to us on restricted terms, and test sets whose release would let a subject train against them are withheld or released in redacted form. Where we withhold, the finding states what was withheld and why, and an independent party may request access under the terms set out in the report.