BEIS — Enterprise Intelligence Standard
An open specification for measuring enterprise intelligence systems: retrieval, citation integrity, refusal, permission correctness, freshness, speed, oversight. Every metric names how it gets gamed, and we publish our own score including where we score nothing.
An open specification for measuring enterprise intelligence systems, with a reference harness anyone can run. Version 0.1, draft for comment.
Version 0.1 and draft, not 1.0. One organisation publishing a specification is a proposal, not a standard — it becomes a standard when somebody who does not work for us runs it and disagrees with the result. Numbering it 1.0 today would be claiming an adoption that does not exist.
The rules this specification is written under
Most vendor-published standards define the metrics their author wins on. These are the constraints that push against that, and they are enforced by tests rather than by intention:
- A metric is in the standard only if it can be measured by a THIRD PARTY without our cooperation. Anything requiring our internal telemetry is marketing dressed as a metric.
- Every metric names its failure mode: what a system that games this metric would look like.
- We publish OUR OWN score for every metric, including the ones we score badly on. A standard whose author scores full marks is a sales sheet.
- Capabilities we do not have are still IN the standard, scored as not-implemented for us. Omitting them because we would score zero is the most common way a vendor benchmark lies.
- No metric ranks named third parties. We cannot run their systems, and a comparative score we invented would be fabrication.
We currently score "not implemented" on 3 of 11 metrics. They stay in the standard. Dropping a metric you would score zero on is the most common way a vendor benchmark lies, and it is invisible to the reader.
The metrics
Retrieval accuracy @1
For a question with a known-correct source, is that source the FIRST one cited.
How to measure it. A fixed question set written from the corpus, with the expected source chosen by reading the page rather than by running the system. Score = correct firsts / questions.
How it gets gamed. Reporting @3 or @5 only. Both are legitimate measures and both are more flattering; a system that puts the right answer third is not the same as one that puts it first, because the reader starts at the top.
BvLogic scores 66.7%. Recovered from 58.3% after the cause was found and fixed: our own synthesis pages were outranking the sources they quote, so they are now weighted down in the index. The dip and the recovery are both published.
Retrieval accuracy @3
Is the known-correct source anywhere in the first three citations.
How to measure it. Same question set, first three citations.
How it gets gamed. Widening the citation set until everything is in it. Report the set size alongside the score or the number means nothing.
BvLogic scores 100%.
Citation integrity
Does the page actually cited contain the answer, or merely rank well for it.
How to measure it. Check the distinctive terms of the question against the text of the cited page. A weak but honest proxy for whether the citation is real.
How it gets gamed. Citing the largest page on the site, which covers many topics and is therefore rarely wrong in a checkable way.
BvLogic scores 88.7%.
Refusal accuracy
Does the system decline when the corpus contains no answer.
How to measure it. Questions with no published answer, including plausible-sounding ones in adjacent domains. Score = declined / should-have-declined.
How it gets gamed. Nothing, which is why it is the most important metric here and the one most often absent. Any retrieval system can find something. The ones that damage a business are the ones that always find something.
BvLogic scores 75%. We fail on questions where every individual word is common in the corpus but the combination is unpublished. Lexical retrieval has no signal for that, and we have not solved it.
Permission correctness
Whether access decisions hold under adversarial conditions, not happy-path ones.
How to measure it. A named case list, each case a specific failure: a stale access list, a group revoked mid-session, deny-versus-allow precedence, nested and cyclic groups, cross-tenant leakage, result-count disclosure, and an answer drawing on a source the asker cannot see.
How it gets gamed. Testing only the happy path. Anyone can make 'the finance team can see the finance folder' work; the leaks come from the seven cases above.
BvLogic scores 100% of 30 cases.
Index freshness
How old the index answering the question is, not how often a crawler is scheduled.
How to measure it. Read the build stamp from the live answering system, not from the build machine. Those diverge the moment a build is not deployed.
How it gets gamed. Quoting the crawl schedule. A daily crawler that has been failing for a week is still described as daily.
BvLogic scores under 1 hour.
Response time
p50 and p95 end to end, over the public internet, at the rate limit a normal user faces.
How to measure it. Measured from the client, including network. Not server-side processing time.
How it gets gamed. Reporting server-side time, or measuring with the rate limiter disabled. A benchmark that needs the limiter turned off is measuring a system nobody uses.
BvLogic scores p50 593ms, p95 646ms.
Cross-document reasoning
Whether the system can answer a question that requires combining two or more sources.
How to measure it. Questions whose answer exists only as the combination of two documents, with the combination verified by hand.
How it gets gamed. Any generative system will produce a fluent answer to these whether or not it is correct, so this metric is meaningless without the citation-integrity metric applied to every claim in the answer.
BvLogic scores not implemented. Our engine retrieves and quotes; there is no synthesis step. It cannot do this, so it scores zero rather than being omitted.
Fabrication rate
Proportion of factual claims in an answer that are not supported by a cited source.
How to measure it. Sample answers, extract each factual claim, check each against the cited passages by hand.
How it gets gamed. Counting only claims the system chose to cite. The claims that matter are the uncited ones between the citations.
BvLogic scores not applicable. No text is generated, so nothing can be fabricated. This is a property of a deliberately limited design, not an achievement, and reporting it as 0% alongside systems that generate prose would be misleading.
Action safety
For systems that act: whether every action is authorised, reversible or confirmed, logged, and scoped to the asker's own permissions.
How to measure it. Attempt actions as an identity that should not be permitted to perform them; attempt actions whose blast radius exceeds the stated scope.
How it gets gamed. Demonstrating a confirmation dialogue. The question is what happens when the confirmation is automated away by a customer under time pressure, which every customer eventually does.
BvLogic scores not implemented. We have built no action layer. An action layer on an unproven permission model is how a product deletes a customer's resources.
Human oversight
Whether a human can see why the system produced an answer, and override it.
How to measure it. Does the system expose its retrieval trace, its scoring, and the terms it did not understand.
How it gets gamed. Showing a confidence score. A number with no derivation is not oversight.
BvLogic scores implemented. Ask BvLogic exposes tokenisation, term weights, the pivot term, candidate counts and the terms it did not recognise.
Deliberately not in the standard
A specification is judged as much by what it declines to measure. These are excluded with the reason, rather than left out quietly:
| Not measured | Why |
|---|---|
| Agent reliability | We have no agents, and a metric written by someone with no implementation is a guess dressed as a specification. It belongs in the standard when somebody with a working system proposes how to measure it. |
| AI decision quality | Requires a ground truth for what the right decision was, which is usually only knowable years later and is confounded by everything else the organisation did. |
| Cost per query | Measurable, but not comparable across architectures. A retrieval system with no inference has near-zero marginal cost and worse capability; reporting it beside a generative system's cost would flatter us for a limitation. |
Run it yourself
The harness and question set behind our own numbers are ours to share. A result you produce is worth more than one we publish, and disagreement is the point — a specification only becomes a standard when somebody who does not work here runs it and argues with the outcome. Ask us for it, and see our full results including the cases we fail.
Educational and informational. Not legal, financial or investment advice.
Everything here is free and stays free. There is no form in front of any document. If you want to know when new guides and templates go up, leave an email.
Roughly monthly. Unsubscribe in one click. We do not share your address, and we will not call you.