BEIS Conformance Statement
How to publish a BEIS result so a reader can trust it: every metric measured including the bad ones, every metric that could not be measured with its reason, and a validator that rejects a cherry-picked statement.
Version 0.1, draft, published 2026-08-12. How to publish a BEIS result so a reader can trust it, and how to check one you were handed.
A conformance statement carries EVERY metric the harness measured, including the ones scored badly, plus every metric it could NOT measure and why. Omitting a metric invalidates the statement. There is no field for 'metrics we chose to include'.
Why a format at all
The harness lets somebody measure their own system. Nothing yet lets a reader trust what they publish about it. A vendor can run a conformance tool, take the three metrics that flattered them, print those on a slide, and be telling the truth in every individual number while the overall impression is false. That is not a hypothetical failure mode: it is how almost every published benchmark result in enterprise software is assembled.
So the statement format is built around one idea: you publish the whole run or you publish nothing. A partial statement is invalid, and the validator says so.
What a statement must carry
| Field | Why |
|---|---|
system | What was measured, named specifically enough that a reader could ask you about it. |
date | A measurement describes a moment. An undated conformance claim is unfalsifiable, because any later disagreement can be attributed to change. |
beis_version | Which version of the specification. An implementation cited against no version cannot be compared with anything. |
question_set | Where the questions came from and how many. A result on questions the vendor wrote about their own strengths is a different claim from one on a customer's questions, and the reader must be able to tell which they are looking at. |
refusal_cases | How many questions the system SHOULD have declined. Zero is a valid answer and it caps what the statement can claim: a run with no refusal cases measured confidence, not accuracy. |
measured | Every metric the harness produced, in full. Not a selection. |
not_measured | Every metric that could not be measured, each with the reason. This is the field a flattering statement would omit and the one a reader should look at first. |
run_by | Who ran it: the vendor, the customer, or a third party. It does not invalidate a self-run result to say it was self-run; concealing it does. |
What makes a statement invalid
Not weaker. Invalid. The validator exits non-zero on each of these.
- Any metric the harness produced that is absent from `measured`. The statement is not partial, it is invalid.
- A `not_measured` entry with no reason. 'Not measured' without a why is indistinguishable from 'measured badly'.
- A question set described only as 'internal' or 'standard'. If a reader cannot tell where the questions came from, they cannot weigh the result.
- A date more than twelve months old presented as current, without a re-run.
- Any aggregate 'overall score' derived from the metrics. BEIS has no overall score, and inventing one is how a bad metric gets averaged away.
Reading somebody else's statement
What a reader should do with somebody else's statement. Published because a format only helps if the people receiving it know where to press.
- Look at `not_measured` before `measured`. Four metrics unmeasured is a different system from four metrics scored well.
- Check `run_by`. A self-run statement is not worthless, but it is a different weight from a customer-run one.
- Check `question_set`. A vendor-written set on a vendor's own corpus is close to a self-graded exam.
- Check `refusal_cases`. If it is zero, ignore the accuracy figures: nothing tested whether the system knows what it does not know.
- Re-run it yourself. The harness is MIT-licensed and the questions can be your own. A result you produce beats any statement you are handed.
Ours
BvLogic publishes its own statement under the same rules, including the retrieval figure that fell to 58.3% before it recovered. A format whose author exempts themselves is not a format.
| System | BvLogic Ask, retrieval over published content, bvlogic.com |
| Date | 2026-08-11 |
| Run by | vendor (BvLogic) |
| Question set | 16 questions, written by BvLogic against its own corpus. Self-authored, which a reader should weigh accordingly. |
| Refusal cases | 4 |
retrieval-1 | 66.7% |
retrieval-3 | 100% |
citation | 88.7% |
refusal | 75% |
permission | 100% (30 of 30 adversarial cases) |
speed | p50 565ms, p95 606ms |
reasoning | not measured — no synthesis layer exists; retrieval only |
hallucination | not measured — no model in the request path to hallucinate |
action-safety | not measured — the system takes no actions |
oversight | not measured — no approval flow to observe |
Three of the eleven BEIS metrics are scored 'not implemented' in our own specification rather than dropped, and the retrieval figure was published at 58.3% for a period before the cause was found and fixed. Both are in the record.
Produce and check one
python beis_conformance.py --endpoint https://your.system/ask --questions your-questions.json --statement statement.json --system "Your system" --run-by customer --question-source "written by us, 40 questions"
python beis_conformance.py --validate someone-elses-statement.jsonThe validator is the half that matters. Emitting a statement about your own system is easy and self-serving; being able to check one you were given is what makes the format worth having. It is in the same MIT-licensed file as the harness.
Licence. This specification is published under CC BY 4.0: implement it commercially, republish it, adapt it, with attribution and a note of what you changed. The conformance harness is MIT. Full terms and what is not covered: the licence page.
Everything here is free and stays free. There is no form in front of any document. If you want to know when new guides and templates go up, leave an email.
Roughly monthly. Unsubscribe in one click. We do not share your address, and we will not call you.