Why my security scanner refuses to tell you your code is clean

The failure mode nobody instruments for

There is a failure mode in security tooling that almost nobody instruments for, and once you see it you cannot unsee it: an empty findings list looks identical to a clean result.

A scanner that crashed halfway, a scanner whose API key expired, a scanner that ran out of budget, and a scanner that genuinely examined everything and found nothing — all four produce the same artifact: findings: []. The CI job goes green. The PR merges. Nobody involved can tell which of the four happened.

I built an application-security agent, and this is the problem I ended up designing the whole thing around.

What it does instead

$ pip install sechelix
$ sechelix audit examples/demo-app

  BLOCKED    authorization   no reasoning executor configured; this node
                             analyses code and cannot be answered by the
                             runner alone
  SUCCEEDED  map
  BLOCKED    verify          dependency not satisfied: authorization
  BLOCKED    gate            dependency not satisfied: verify

RESULT  INCOMPLETE - unsatisfied mandatory nodes: gate, verify
        No security claim can be made from this run.

Exit code 1. Not because your code is bad — because nothing was examined, and the tool will not pretend otherwise.

The states are deliberately distinct:

SKIPPED and BLOCKED are the pair that matters. An inapplicable lane is a real answer. An unaffordable verifier is not.

The budget case is the one that convinced me

Give a run a cost ceiling and let it run out just before the verifier:

map     SUCCEEDED
authz   SUCCEEDED
verify  BLOCKED    max_cost_usd budget exhausted: requested 0.5, 0.1 remaining
gate    BLOCKED    dependency not satisfied: verify

unsatisfied mandatory: ['gate', 'verify']

A budget limit that silently skips verification and then returns PASS has converted a cost control into a correctness bug. That is strictly worse than having no budget at all, because now the failure is invisible. There is a test that starves the budget mid-verification and asserts the gate never passes.

Blinding the verifier

The second design decision: a hunter proposes findings, and an independent verifier tries to refute them. The verifier is not told the hunter’s confidence, severity, verdict, or exploitability — those fields are stripped before the candidate is handed over.

The reason is mundane. A verifier that reads “HIGH confidence SQL injection, definitely exploitable” before it looks at the code is not verifying. It is agreeing.

I tested this by planting a fake finding. Two candidates went in: one real, against a file with a genuine missing ownership check, and one fabricated against that file’s clean twin, tagged as loudly as I could make it — confidence: HIGH, severity: CRITICAL, verdict: exploitable, hunter_notes: certain this is a real IDOR.

The verifier kept the real one and refuted the plant:

refuted: the candidate locating the missing-ownership claim at
clean/orders.py:get_order is false. The clean variant explicitly compares
order['user_id'] to session['user_id'] and returns (None, 404) on
mismatch before the success return.

It named the exact line. And a capture of the prompt actually sent confirms HIGH, CRITICAL, exploitable and the hunter’s note never reached it.

Paired fixtures, because precision is the hard part

The demo corpus has five vulnerable files and five clean counterparts, each pair differing in exactly one security-relevant way.

This is not decoration. A corpus that only contains bugs cannot measure whether a tool stops. An agent that flags every line scores perfectly on a vulnerable-only benchmark and is useless in a real repository. If the clean twin is not in the corpus, a false positive is invisible.

It found a bug I wrote

I pointed the tool at its own runner. It found a path traversal: run_id arrives from the command line — sechelix report <id> — and was joined straight onto the runs directory.

run_id='../../outside.json'  ->  resolves outside the workspace

report would then read that file and print it. It was fixed with a shape check plus a resolved-path confinement check, and ten regression tests. It is in the history at commit 0e50b56 rather than quietly patched.

What I have not measured

There is a blind evaluation: 76 cases judged by 76 independent processes, each launched from an empty directory containing only the case file — precision 0.950, false-positive rejection 0.947. That is a label-only result on 38 authored pairs. It is not a measurement of the full workflow, which is still marked NOT_MEASURED in the repository. The raw result is evals/results/claude-sonnet-5-blind-2026-09-02.json, and the research index explains what it does and does not cover.

I also have not benchmarked against SEC-AF, Cloudflare’s security-audit skill, or Strix. So I am not claiming to beat them. The comparison does not exist yet, and saying so is cheaper than being caught.

Try it

# Agent Skill — Claude Code, Codex, Copilot. No Python.
npx skills@latest add omarmohelal/SecHelix --skill sechelix

# Optional runner
pip install sechelix
sechelix doctor
sechelix audit examples/demo-app

STATIC is the default and performs no network access at all. No account, no email, no cloud. Apache-2.0: github.com/omarmohelal/SecHelix.