We were asked to prove the engine is good rather than claim it. Proving means someone who does not trust us can check, so this page publishes the method, the question set, the score, and the questions we currently get wrong.

Results

Measured against the live engine at https://bvlogic.com on 2026-08-10. Every figure below is generated from the run — this page has no hand-typed numbers in it.

What we get wrong

A benchmark that only lists passes is advertising. These are the cases the engine fails today:

Why the refusal failure is hard. Asked for a private internal figure, the engine answers from a published market report. Every individual word of the question is common in our corpus, so there is no term-level signal that the combination is something we have never published. Lexical retrieval cannot detect that, and we would rather say so than add a keyword blocklist that makes this one test pass while the underlying weakness remains.

Method

1. A fixed set of 16 questions, written from published material. The expected source for each was chosen by reading the page, not by running the engine and recording whatever it returned — a benchmark whose answers come from the system under test measures nothing.
2. Questions that should be refused are included and weighted as heavily as the rest. Any retrieval system can find something; the ones that damage a business are the ones that always find something.
3. The harness runs against the live production engine over the public internet, at the same rate limit as any visitor. It identifies itself as a probe so its own traffic is never counted as customer demand.
4. Both top-1 and top-3 are published, because top-3 alone is true and misleading.

What this does not measure, and why

Rows are absent rather than estimated. Scoring a capability that does not exist is the exact thing a benchmark is supposed to prevent.

The metric definitions behind this page are published separately as BEIS, the BvLogic Enterprise Intelligence Standard — including the metrics we score nothing on, and how each one gets gamed.

Run it against your own system

This section used to say the harness was ours to share and to ask us for it. That was true of the method and not of the code: our own benchmark hardcodes this site, so nobody outside this company could actually run it. An offer that cannot be executed is not an offer.

Download the conformance harness — one file, standard library only, no connection to us, and it sends us nothing.

python beis_conformance.py --endpoint https://your.system/ask \
                           --questions your-questions.json

It scores what can be judged from outside and declares the rest as not measured rather than skipping it. Permission correctness, freshness, action safety and human oversight depend on your access model and your pipeline, which no HTTP response reveals, and a conformance report that quietly omits what it could not test is the failure this specification exists to prevent.

When you have a result, the conformance statement format sets out how to publish it so a reader can trust it, and how to check one you were handed. The harness emits a statement with --statement and validates somebody else's with --validate.

Bring your own questions, and include some the system should refuse. A benchmark with no refusal cases measures confidence rather than accuracy: one that answers everything scores perfectly and tells you nothing. A result you produce is worth more than one we publish, including a result that makes us look worse.