Plain files
Everything is JSON, one line per result, and fingerprints. No database, no special format, nothing to install to read it.
Touchstone
AI evaluations anyone can check
An open tool for running AI tests and recording results in a verifiable evidence bundle, so anyone can check the numbers without installing anything or taking our word for it.
TOUCHSTONE SANDBOX
ready
Edit a pass mark and the verdict moves. Shown as JSON, which a browser parses on its own; the real file is YAML. Above one repeat it assumes a question is answered the same way each time.
Evidence
Result
Try an example
Runs in your browser. Same arithmetic as the tool, checked against it to twelve
decimal places.
What it is
Touchstone runs a test against an AI system and writes down what happened.
It saves the plan it followed, one line for every question it asked, the results it worked out from those lines, and a fingerprint of every file. All of it is plain text.
That folder is the point. Someone who has never installed Touchstone can open it, redo the sums in a spreadsheet, and confirm nothing has been changed since it was written. They do not have to trust the person who ran the test, and they do not have to trust us.
Every number has a range
A result is never a bare percentage. It carries how far the number could move and how many questions it came from.
It can say it does not know
Where the evidence does not clearly pass or fail, Touchstone says so rather than picking the friendlier answer.
The tests report facts
A test says what happened on each question. Touchstone does the sums, and the raw lines travel with the answer so anyone can disagree.
Tests are boxed in
A test runs in a container and can reach only what it said it needed. Nothing else on the machine, and nothing else on the network.
The pass mark is a file
Grades, cut-offs and what each level of access allows you to claim are read from a plain file, so you set them and a reader can see them.
The problem it solves
A score on its own does not say how much evidence is behind it.
Two systems are tested. Both score 94%. One was asked 50 questions, the other 1,000:
| Questions asked | Score | Where the true rate could sit | Backs a 90% requirement? |
|---|---|---|---|
| 50 | 94% | 83.5% to 98.8% | No. It could be running at 85%. |
| 1,000 | 94% | 92.4% to 95.4% | Yes. |
Everyone can tell a restaurant with five reviews from one with five thousand, and then loses that instinct the moment the same thing turns up with a percent sign on it. The longer version of this argument is in The 94% Problem. Touchstone is what happens when you stop writing the argument and build the thing that refuses to print the first row without the third column.
Our bet
Six things we think are true. The tool is what follows from them.
Checking beats trusting. The market for AI testing is full of claims that a system was tested, and almost none of them can be checked by the person being asked to rely on them. A report says a number. A slide says a number. Neither carries the questions that were asked or the answers that came back, so a reader who doubts it has nowhere to go. We think the thing worth building is not another way to produce numbers, but a way to hand someone the working out. Everything else here follows from that.
Built for the person who has to decide. Someone changing a prompt and running it again twenty times does not want files sealed and fingerprinted every round; that is friction with no payoff. Use Inspect for that. Touchstone is for the moment the work stops being an experiment and becomes a claim: a report going to a regulator, a supplier answering a procurement question, a bank's own reviewers checking a model before it goes live. That reader cannot rerun the test. They can only check the record, so the record has to be worth checking.
A number without a range is not a result. Most tools let you print a bare percentage, and so most reports do. We took that option away. Touchstone stores counts rather than rates, so there is no way to hand it a number that has already lost its sample size, and no path through the code that prints one. It is a small rule and it changes what the tool can be used to argue.
Saying "not sure" is a feature. When the range straddles the pass
mark, the evidence genuinely does not say which side the system is on. Every tool we
looked at prints a verdict anyway. Touchstone returns
indeterminate and names the boundary the range crosses. A grade that can
only ever be good news is not a grade, and the ability to withhold one is most of what
makes the others mean anything.
Be loud about what it cannot do. The range covers one kind of error and there are bigger ones it says nothing about. We write those on the tool, in the docs and on this page, because a reader who learns the limits from us can still use the result, and one who finds them later cannot use anything we ever publish. The same rule applies to our own mistakes: when we found that Touchstone had been getting a number wrong, we shipped the fix and wrote down what it had cost.
Small enough to carry. Three outside pieces of code and sums written against Python's own library, so the whole thing installs onto a machine with no internet. The places where this work matters most are often the places with the worst connection, and a tool that needs a fast link to a package server is a tool that is not there when it is needed.
What it does not prove
The range covers one kind of error, and it is rarely the biggest one.
The range answers one question: how far would this number move if you asked a different set of questions of the same kind. It does not answer any of the following, and no amount of fingerprinting makes it.
Limits of this claim
Your questions are not a random sample of real use, so a system can do well here and badly in service. Whatever decided an answer was right or wrong has its own error rate, and when that judge is another AI its mistakes come in patterns rather than at random. And if the questions leaked into the system's training, you are measuring memory rather than ability. Freezing a plan and fingerprinting a folder fix the record and the arithmetic. They do not make the questions the right ones. It is precision, and it is not accuracy.
A grade from Touchstone is evidence for a decision. It is not an approval, and nothing it produces licenses or certifies anything.
How a run goes
Seven commands, in order.
Only one of them needs a container. Everything after it just reads files, so the last step works on a plane with the wifi off.
| Command | What it does |
|---|---|
validate | Checks the plan against what each test says it needs. |
freeze | Pins the exact versions, fixes the random seeds, fingerprints the plan. |
run | Runs the tests and writes one line per question. |
estimate | Works out the rates and their ranges, split by whatever groups you asked for. |
grade | Applies your pass marks and grades each measure. |
bundle | Fingerprints every file and writes the list. |
verify | Re-checks a folder against that list, offline. |
verify is the one a stranger runs, so it is the one everything rests on.
It needs no network, no container, and no cooperation from whoever produced the folder.
Get started
It is free and open.
Touchstone is Apache 2.0 and the issue tracker is open. It is early: it works and is tested doing it, and the pass-mark file format is still moving.
If you are producing evidence someone else has to rely on, or you have to rely on evidence somebody handed you, we would like to hear which part of this is missing.