Evaluation Methodology

The required record

SecHelix does not claim benchmark performance until a result is reproducible. docs/EVALUATION.md defines the minimum evidence required before publishing any accuracy, recall, false-positive, or coverage number.

Every published run must record:

• SecHelix commit or version.
• Fixture or target commit or version.
• Environment and execution mode.
• Model, provider, and version where available.
• Agent host or client.
• Enabled scanners and tools, with versions.
• The prompt or instruction used.
• Deterministic configuration and seeds where available.
• Start and end timestamps.
• Expected labels.
• Observed labels.
• Verifier outcome.
• Report and retest artifacts.
• Known limitations and blocked checks.

The six metrics

Verified precision

Verified true positives divided by all reported verified findings. A candidate rejected by the verifier is not a verified finding and stays visible in the rejection accounting.

Detection recall

Verified expected vulnerabilities found, divided by total verified expected vulnerabilities in the fixture set. Only fixture sets with documented ground truth qualify.

False-positive rejection rate

False candidates correctly rejected, divided by all intentionally false candidates presented. This measures the central claim directly.

Applicability accuracy

Correct APPLICABLE, NOT_APPLICABLE, UNKNOWN, and BLOCKED classification. UNKNOWN and BLOCKED are not collapsed into success.

Regression-proof rate

For verified findings that received a fix, the fraction with a test that fails before and passes after, plus a successful retest.

Release-gate accuracy

Whether the final PASS, PASS_WITH_KNOWN_RISK, BLOCKED, or INCOMPLETE decision matches the documented evidence policy.

Minimum first benchmark

The first measured release does not need a large corpus. It needs honest ground truth and reproducibility. The recommended minimum is one clean and one vulnerable case for each of:

• Authorization.
• Injection or dataflow.
• State machine or business logic.
• Race or idempotency, where deterministic reproduction is possible.
• Secrets or supply chain.
• AI, agent, or MCP, if claiming runtime AI-security evaluation.

Each case should be non-trivial enough that a keyword match alone cannot solve it.

Real repositories are kept separate from synthetic scores. A public case study must include authorization to test, the repository and commit, the relevant SecHelix commit, a safe description of the failed boundary, the evidence trail, the fix reference when public, regression proof, attribution permission, and an explicit statement if the finding was previously known.

Comparison policy

No better-than-tool-X claim is published unless all six conditions hold:

• The same targets and ground truth are used.
• Tools receive comparable access.
• Metric definitions are identical.
• Configurations and versions are recorded.
• Raw results can be independently inspected.
• Limitations are stated.

Scanner finding counts are not an accuracy metric.

Reporting template

SecHelix commit:
Target/fixture version:
Host/model:
Mode:
Tools:
Cases:
Expected vulnerable:
Expected clean:
Verified TP:
Verified FP:
False candidates rejected:
Verified FN:
Unknown:
Blocked:
Verified precision:
Detection recall:
FP rejection rate:
Regression-proof rate:
Release-gate accuracy:
Artifacts:
Limitations:

One run satisfies this protocol today: the blind label suite result recorded on 2026-09-02. It is label-only, so applicability accuracy, regression-proof rate and release-gate accuracy remain NOT_MEASURED until a run exercises the full workflow end to end.