What has been measured, and everything that still has not.
An index of the published evidence: the first uncontaminated blind label run and its limits, the full-workflow metrics that remain NOT_MEASURED, the eval lab and blind packet, the single case study, the self-audit, the discovery baselines, and the methodology references.
What is on this page
This is an index, not a paper. Every item below points at a file that exists in the public framework repository, and the wording here is meant to be no stronger than the wording in the file it links to. Where a document is a dated snapshot, it says so, and the tree has moved since.
The single most important entry is the first one, because it is the one that separates the one number that exists from the many that do not.
Blind label suite — MEASURED; full workflow — NOT_MEASURED
The evaluator disqualification that kept this unmeasured for months is deliberately broad, and still applies to any future run: any session that has read evals/fixtures/, scripts/build_eval_fixtures.py, or gold-packs/*/pack.json is excluded, because partial knowledge of a paired suite still leaks the answer to its twin.
What the published run carries, and what any replacement must also carry:
- A model or session that has not read the fixtures, the fixture builder, or the Gold Pack definitions.
- The blind packet evaluated in an isolated session, with the packet digest recorded.
- A prediction packet with every field of the run record populated.
- Scoring through the committed harness rather than by hand.
- Publication only if the SecHelix commit, agent host, model, and limitations are all present — whatever the number turns out to be.
The evaluation lab and the blind packet
The lab holds synthetic, non-destructive, paired controls: a vulnerable fixture and a clean sibling that looks alarming but is protected by a real control. The pairing is the point — it measures rejection as well as detection. No model or scanner is called by the repository itself.
| Inventory | Value |
|---|---|
| Fixtures | 38 |
| Cases | 76 |
| Distinct family labels | 10 |
| Blind packet | 76 cases, ground truth withheld until scoring |
| Blind packet checksum | c15861edbe997f5909d32d3a0870eb963b3c654b37583aaa6ba28d90d183d9fb |
- Blind packet
evals/blind-packet/contains the 76 blind cases and a one-command procedure written for a fresh session. Ground truth is not in that directory; it lives in the fixtures and is read only at scoring time.- Eligibility
- The run procedure opens by disqualifying anyone who has already opened the fixtures, the fixture builder, or any Gold Pack definition. A contaminated run is treated as worse than no run, because it produces a number that looks real.
- Harness status
- The scoring harness is validated. A naive keyword matcher was scored end to end to prove the harness works and to evidence fixture difficulty. That result is flagged
is_sechelix_result: falsein its own file, and its numbers are deliberately not reprinted here so they cannot be mistaken for a SecHelix score — read them in the file if you want them.
The published case study
One end-to-end run against a real application rather than a synthetic fixture, published because the outcome is useful in both directions: one finding was verified and fixed, and one plausible high-severity candidate was refuted during independent verification. It was an owner self-audit of a private ~600-line storefront, run in STATIC plus LOCAL mode with zero scanners enabled and nothing outside the local machine contacted.
The most useful section is the one about the process rather than the application: a stale build cache nearly certified a fix that had not actually shipped. That is recorded in the case study instead of being edited out.
The self-audit
SecHelix reviewed its own V3.3 change set using its own modules, in STATIC mode, with no scanners enabled and nothing contacted over the network. The differential reviewer returned 17 NEW_RISK deltas and 6 UNKNOWN, and none of them was promoted to a finding — which is what the module contracts for.
The value of the run was elsewhere. The first pass flagged the project's own prose: a code comment, a docstring body, and a JSON Schema description field, none of which do anything. That is precisely the false-positive class the project exists to reject, appearing inside the project's own reviewer. It was fixed — non-secret rules are now suppressed on commentary across several comment syntaxes, docstring bodies, and prose-valued JSON keys, while a credential pasted into a comment is still reported, because a credential in a comment is still a credential.
Read it: docs/research/self-audit-v3.3.md.
Discovery baselines
Discoverability work is otherwise unfalsifiable: without a recorded starting position, any later claim of improvement is unverifiable. So the starting position was recorded before it was flattering. The pre-launch baseline scored 0 of 6 — SecHelix was not discoverable by any tested query, including its own name.
- Pre-launch baseline
- Six queries run once each on a recorded date, from the brand-name query down to the differentiator query. Every row NOT_FOUND. The zero point.
- Release-day measurement
- A second pass taken within an hour of publishing, explicitly labelled a starting line rather than a verdict, because indexes and crawlers do not update in an hour.
- Recurring measurement
- An append-only log. Blocks are added, never overwritten, because the value is entirely in the comparison.
- RATE_LIMITED is not NOT_FOUND
- A refused API request is recorded as RATE_LIMITED. An error is not a measurement — an earlier pass read rate-limit responses as absence until the responses were actually inspected, and that mistake is documented rather than quietly corrected.
Methodology references
The documents that define what SecHelix would have to demonstrate before any number is published, and the contracts that a published run has to satisfy.
Standing reports
Longer write-ups from individual work sessions. Each is a dated snapshot: the numbers in them were true on the date at the top, and the tree has moved since.