Benchmark Lab
One layer is measured. The other is not, and the difference matters.
The blind label suite has one uncontaminated run. The SecHelix workflow — applicability, the independent verifier, remediation with regression proof, the release gate — has never been measured end to end. Both statements are on this page, in that order, because only one of them comes with numbers.
Blind label suiteMEASURED
Full workflow: NOT_MEASURED.
Layer A · blind label suite
MEASUREDOne uncontaminated label-only run
On 2026-09-02, 76 cases were judged by 76 separate processes. Each was launched from an empty directory holding only the case file. None cloned the repository, and none saw a label, a rationale, a pairing, or how many cases were vulnerable. Each was asked one question about one file and returned one label.
Counts: TP 38 · FP 2 · TN 36 · FN 0. The two false positives were one Business Logic / Payments case and one SSRF / URL Fetching case; every other family scored zero. Committed as evals/results/claude-sonnet-5-blind-2026-09-02.json.
For scaleDeterministic regex floor
Precision0.511
Detection recall0.632
False-positive rate0.605
FP rejection rate0.395
A keyword matcher scores at chance here. That is a statement about fixture difficulty, not a comparison SecHelix wins. It is committed separately as baseline-keyword-v1.json with result_kind: HARNESS_BASELINE and is_sechelix_result: false.
What these four numbers do not say
They describe a single-pass label judgment on an authored suite. They are not “SecHelix accuracy”, they are not a detection rate on production code, and no comparison to any other tool is offered or implied.
- One model, one run. No repeats, no seed control, no variance estimate.
- A balanced 38/38 authored suite. Real code is overwhelmingly clean, so precision here overstates precision in the field.
- Mostly single-file and mostly Python. Real vulnerabilities often span modules.
- Synthetic fixtures, authored rather than harvested from incidents.
Every limitation is recorded in the result file itself, not only here.
Layer B · the full workflow is NOT_MEASURED
The protocol behind Layer A asks for one label per file and takes one label back. It never ran attack-surface mapping, the independent refutation pass, the evidence adapters, evidence-chain construction, remediation, regression proof, or the release gate — which is most of what SecHelix actually is.
Verified precisionNOT APPLICABLE TO THIS RUN
Applicability accuracyNOT_MEASURED
Regression-proof rateNOT_MEASURED
Release-gate accuracyNOT_MEASURED
Why verified precision is not a score: the run recorded verification_status: NOT_RUN for all 76 cases, so the harness computes 0.0. Nothing was independently verified because nothing was asked to be. The three remaining metrics stay the literal string NOT_MEASURED rather than a misleading zero.
Fixture suiteAvailable
evals/fixtures/Paired vulnerable and clean cases
Ten families: authorization, authentication, injection, browser boundary, SSRF, files, business logic, race conditions, supply chain, and AI/MCP. Each case is a realistic module rather than a snippet, so a keyword match alone cannot resolve it.
- Fixtures
- 38
- Paired cases
- 76
- Families covered
- 10
- Case size
- 38–66 lines
Measurement protocolHow the run was produced
- 01Export blind cases
Case identifiers are opaque digests. The variant, fixture id, and rationale are withheld.
- 02Predict without the corpus
The evaluated session must not have authored or read the fixtures. One process per case, empty directory.
- 03Score deterministically
The harness maps opaque ids back to ground truth and computes every metric.
- 04Publish with provenance
Commit, packet digest, model, host, mode, tools, and limitations are all recorded — whatever the number turns out to be.
The record supports MEASURED, INSUFFICIENT_SAMPLE, and BLOCKED without inventing values.