Retrieval Benchmark
Our published benchmark results, the methodology behind them, and the cases we currently fail.
We were asked to prove the engine is good rather than claim it. Proving means someone who does not trust us can check, so this page publishes the method, the question set, the score, and the questions we currently get wrong.
Results
Measured against the live engine at https://bvlogic.com on 2026-08-10. Every figure below is generated from the run — this page has no hand-typed numbers in it.
| Measure | Result | What it means |
|---|---|---|
| Retrieval accuracy, top 1 | 66.7% | The first source cited is the right one. 12 questions. |
| Retrieval accuracy, top 3 | 100% | The right source appears in the cited set. This is the flattering number; we publish both. |
| Refusal accuracy | 75% | Declines when we have published nothing. 4 questions we should not answer. |
| Citation coverage | 88.7% | The cited page actually contains the terms of the question asked. |
| Permission accuracy | 100% | 30 adversarial cases: stale permissions, revoked groups, cyclic groups, cross-tenant access, count leakage. |
| Response speed | 585ms / 625ms | Median and 95th percentile, measured end to end over the public internet. |
| Index freshness | 0.1 hours | Age of the index that answered these questions. 4,093 passages. |
What we get wrong
A benchmark that only lists passes is advertising. These are the cases the engine fails today:
| Question | Expected | What happened |
|---|---|---|
| What is the BvLogic CEO salary? | a refusal | answered: /research/ai-salary-report/ |
Why the refusal failure is hard. Asked for a private internal figure, the engine answers from a published market report. Every individual word of the question is common in our corpus, so there is no term-level signal that the combination is something we have never published. Lexical retrieval cannot detect that, and we would rather say so than add a keyword blocklist that makes this one test pass while the underlying weakness remains.
Method
- A fixed set of 16 questions, written from published material. The expected source for each was chosen by reading the page, not by running the engine and recording whatever it returned — a benchmark whose answers come from the system under test measures nothing.
- Questions that should be refused are included and weighted as heavily as the rest. Any retrieval system can find something; the ones that damage a business are the ones that always find something.
- The harness runs against the live production engine over the public internet, at the same rate limit as any visitor. It identifies itself as a probe so its own traffic is never counted as customer demand.
- Both top-1 and top-3 are published, because top-3 alone is true and misleading.
What this does not measure, and why
Rows are absent rather than estimated. Scoring a capability that does not exist is the exact thing a benchmark is supposed to prevent.
| Not measured | Reason |
|---|---|
| Cross-document reasoning | The engine retrieves and quotes; there is no synthesis layer to score |
| Agent execution | No enterprise agents run in production today |
| Cost per query | No model inference in the request path, so there is no per-query model cost |
| Competitor scores | We cannot run their systems. Publishing numbers for software we have not run would be fabrication, and we will not do it to win a comparison table |
If you run enterprise search and want to compare properly, the question set and the harness are ours to share — ask us and run it against your own system yourself. A comparison you run is worth more than one we publish.