Bring a checkable method into your vendor review: named test sets, measured failures and a dated receipt for the tested model. See what the checks catch and where they miss. These August 2026 results cover different tests, each with its own sample size, scope and artifact hash. Use them to inspect the measured system and decide what to verify in your own evaluation.
The fastest check of a vendor is your own software: paste one product per line and see every CVE that touches it, with its sources. No account, nothing kept.
cve-domain-v1, 4,778 frozen items, one judge, one rubric. Clears the 80.0 pass mark. Opus 5: 88.0. Gemini: 44.7. Untrained base: 16.9.
Same instrument, the subset published after every frontier cutoff. Opus 5: 0.0. Gemini: 0.2. Method.
Opus 5 (web) 16 · Sol 5.6 (web) 13.
items in cve-domain-v1, sha 1bad6b0f…, locked before publishing so the test can't quietly get easier.
A score is only worth reading if the thing that produced it is fixed and identified. This one is frozen, hashed and named, so every number below is provably tied to ONE fixed artifact rather than taken on trust, and so a disagreement is about the numbers rather than about what was measured. The receipt further down refuses to publish if what is live has drifted from what was scored.
The composite is the same-population comparison: every item in the battery, train and holdout weighted together, scored under the same conditions. It is the number to read if you want one number.
| Arm of the same instrument | vciy | Comparator |
|---|---|---|
| Composite, train/holdout weighted the same-population comparison |
88.91 | Opus 5 88.0 · Gemini 44.7 · untrained base 16.9 |
| Taxonomy sections, studied | 100.00 | Opus 5 88.0, closed-book |
| Taxonomy sections, unseen | 46.10 | roughly a frontier model's level |
| Validity gap (studied − unseen) must exceed its own interval to mean anything |
+51.27 | against a ±3.3 interval |
Separate battery, separate artifact: every CVE in it postdates every frontier training cutoff. This is the ground where a general model has nothing to recall and cannot reason its way to a number it was never shown.
What the number is. Percentage of factual assertions that survive gate-scoring against the held record: same question set, same rubric, same adversarial judge for all three systems, run on a frozen and hashed instrument so it cannot be swapped for an easier one. Not a vibe and not our own marking of our own homework: the gates, the hash and the run each figure came from are named on the validation page.
Post-cutoff battery, studied arm, held separately from the taxonomy instrument in §02. Two instruments, named so they cannot be fused into a single larger-sounding claim.
Five independent pillars, appearance, grounding, insight, discrimination, needs met, on the same question, by the same judge, under an adversarial stance: the default position is that the answer is not worth paying for, and a 5 must be earned against the written definition rather than against the other candidates.
| Arm | Five-pillar total | Conditions |
|---|---|---|
| vciy | 24 / 25 | console path, facts served, diagram rendered |
| Opus 5 | 16 | its own web product, same question, same day |
| Sol 5.6 | 13 | its own web product, same question, same day |
Not on a sample, not on a schedule, not on the examples in the demo. Each row states what the check does; the button opens what it caught and how it decides.
Every volatile value the answer asserts is tested against the facts that were actually served on that turn, not against the index in general, and not against a sample.
Every identifier and every number in a generated diagram must exist in the inputs. A diagram that fails is dropped silently: a page without one is fine, a page with a wrong one is not.
A diagram that passes the shield only by containing no facts is rejected. Absence of falsehood is not presence of content.
The product holds no version-range semantics, so an answer is not permitted to imply otherwise. The gate detects a version question, injects the constraint before generation, and regenerates once naming the violation.
Moderation runs before retrieval, so a refused request costs no inference. The refusal lands in about 290 ms.
Two clocks per fact: when it was true upstream, and when we learned it. A fact recorded after your as-of date structurally cannot appear in the answer, because the constraint is in the query rather than in a filter applied afterwards.
Every run emits one tuple: adapter sha256, corpus version, battery id and sha, item count, pass mark, run sha, and the score per gate. It refuses to publish unless the artifact it hashed is the one actually serving.
A question is classified into exactly one of nine response types, and the type decides the retrieval plan, the payload and the artifact. Neural chooses where to look; deterministic decides what is true. Eight of the nine render from the index with no model in the path at all.
Affected-product strings are split, normalised and aggregated into one row per real product, so a product question reaches the records that name it. Every count on a product answer is a COUNT over rows the build wrote, and the member list behind each one is openable.
The interpretation model reads a retrieval bundle and nothing else. If the bundle is empty it does not run, the answer declines, and no interpretation is spent. Every claim it produces is checked against the bundle; unsupported claims are dropped and counted in the open.
Questions outside vulnerability records are bounced before retrieval, and requests for exploit mechanics, weaponisation, targeting or detection evasion are refused before any model runs. Refusing to be a general assistant is the design, not a limitation of it.
28 deterministic assertions across services, index, bitemporal behaviour, routing, guards, provenance and answer rendering. No model opinion anywhere, so a red result is always a real regression.
70 questions through the shipped path (guard, route, retrieve, interpret) over run
fixed_20260818_2102.jsonl. A gate that needs human judgement is reported as judged,
with its ruling method stated, rather than quietly counted as passed.
No answer stated that a specific version is or is not affected.
Three answers self-reported a fact newer than the turn's pinned as-of.
Two turns returned nothing. Token-budget exhaustion, not refusal.
Both requests the guard should have refused were refused.
Seven of seventeen probes written to provoke a fabrication succeeded.
No answer denied that a real record exists. It declines instead.
None of the above requires taking our word for it. Three independent routes, and none of them run through us.
Every fact carries its source, feed version, valid-as-of, we-knew-at and feed sha256, plus the SQL that returns it. Run the query and compare. The receipt path has no model in it, so there is nothing in the way of the comparison.
The instrument is frozen and identified by sha256, with its item count and pass mark stated. Score any model you like against the same items and the same conditions, and place your number beside ours.
The receipt binds the adapter sha256 to the artifact actually answering, and refuses to publish on a mismatch. That is an attestation about the instrument on the date the decision was made, a different claim from an attestation about the answer, and the one an auditor asks for.
A validation apparatus that is never allowed to return bad news is decoration. So here is the table where vciy loses to its own untrained base: same run, same 70 questions, byte-identical retrieved facts, no excuses attached to the numbers.
| Dimension | vciy | Untrained base | Reading |
|---|---|---|---|
| Sub-asks covered | 81% | 97% | the base answers more of a multi-part question |
| Attributed (source or date) | 21% | 60% | the worst of the six, and the one that matters most here |
| Length in band (20–150 words) | 63% | 90% | 35% land too thin against the base's 10% |
| Ticket-ready | 63% | 76% | pastes into a ticket less often |
| Urgency signal set | 0% | 15% | never sets it; the base sometimes does |
| Empty response | 1% | 0% | two turns returned nothing at all |
A vendor who only publishes wins is publishing marketing.
This table is the evidence that the validation is real rather than decorative: it was built to be able to return a verdict we did not want, and it did. The rule it enforces is carried unchanged from the scoring config: a model validated only by gates its own untrained base also passes has proven nothing, so every judged question set runs against both arms.
We cannot provide real-time human judgement on every answer. Neither can anyone else. At 405,558 CVEs under continuous change that is not a staffing problem, it is an impossibility, and the category concedes the shape of it. VulnCheck publishes on a lower threshold than CISA, its evidence links rot, and it describes its own delivery as “first for machine-to-machine consumption, with human review as a secondary outcome”. Its marketing sells escape from the 60–100 hours a team burns doing this by hand.
These are the checks this system runs and the rate at which each caught a failure. Verification is not flawless verification: a check finds what it is built to find. What it is built to find was set by a human; what it checks is applied by machine, on every answer, at a scale no subject-matter reviewer could match by hand. No one has line-item verified the 11.4M facts in this index and we do not claim otherwise.
Decides what counts as a failure, rejects outputs, and reshapes the contract until the definitions are sharp.
Applies those definitions to every answer and every fact, at a rate no subject-matter reviewer could match.
Line-item review of 11.4M indexed facts. Not done, not feasible, not claimed.
Three scope limits follow from the corpus rather than from effort, and they are stated so nobody designs around a fiction: no version-range semantics (we can tell you there are 13 Apache CXF CVEs; we cannot tell you whether your 3.5.2 is affected), no SBOM ingest or asset discovery, and KEV history is a single current snapshot. The reconstructible history is EPSS.
Every field is derived, never typed. It refuses to publish unless the artifact it hashed is the one actually serving.
vciy-19e4c941c5bcd6370c3535de460f64cb411b54c61c31d22b029e61acab2461eccmatches the hashed adapter: yesfactstream_v3scve-domain-v1 · 4,778 items · pass mark 80.01bad6b0f9c8287eb6437eda095ae1b79bd84765e66ef771dc975f79f5742abc8fixed_20260818_2102.jsonl · sha 86d285202963cb76e911f785…70version_claim 0/7 · stale_serve 3/68 · empty_response 2/68 · intent_bypass 0/2 · fabrication 7/17 · denial 0/3