Benchmark Status

Current status

The bar is procedural. The protocol requires a reproducible run recording the SecHelix commit, the fixture version, the model and provider configuration, the enabled tools, observed outcomes, and the limitations. The published run carries all of it; anything that does not is not publishable.

The same discipline applies to the specialist mesh: all capability measurements are NOT_MEASURED until the eval lab runs reproducible fixtures, and the role-assignment table is not filled with anecdotal model rankings.

The fixture set

The lab holds 38 paired fixtures. Each carries a vulnerable variant and a clean sibling, so the set covers 76 cases across 10 families. They are synthetic source fragments for static reasoning, with no external target and no live credentials. Several pairs are built so the alarming-looking variant is the safe one, protected by a real compensating control.

A representative pair: the vulnerable invoice lookup returns the record straight from the store, while the clean sibling raises a permission error unless the caller tenant matches. The difference is three lines, which is deliberate — the fixtures test reasoning, not pattern size.

Blind scoring

The runner exports cases without expected labels, and scores only after predictions are fixed. No model or scanner is called by the repository itself.

python evals/run_evals.py --export-cases work/blind-cases.json
# review those cases with an authorized model/scanner pipeline
python evals/run_evals.py --predictions work/predictions.json --output work/result.json

Each prediction supplies a case_id, a predicted_label of VULNERABLE or CLEAN, and optionally a verification status and scanner sources. The runner then reports precision, recall, verified precision, false-positive rate, duplicate-root-cause rate, time, token cost, model and provider identity, and scanner contribution where supplied. Missing operational measurements remain NOT_MEASURED; they are never converted to zero.

A committed keyword baseline exists at evals/results/baseline-keyword-v1.json. It measures how far a naive matcher gets, which is the number any real result has to beat to mean anything.

Scoring rules

• Use paired vulnerable and clean fixtures where practical.
• Do not count an unsupported model suspicion as a true positive.
• Keep the verifier blind to the expected answer where possible.
• Publish fixture provenance and scoring rules.
• Separate static detection from runtime proof.
• Record blocked checks instead of treating them as passes.

What would change it

A published score requires a run that satisfies the full evaluation record and covers the recommended minimum corpus: at least one clean and one vulnerable case for authorization, injection or dataflow, state machine, race or idempotency, secrets or supply chain, and AI or MCP if runtime AI-security evaluation is claimed.

The trophy case follows the same rule and is currently empty by design: it only lists findings with public, attributable evidence and permission to be referenced. A new project is better served by an empty trophy case than by unverifiable claims.