Completion Claim Validation Gate.
A Claude Code Stop hook that intercepts an assistant's "done / fixed / deployed" claims and routes them to a local Ollama model prompted to disprove them — blocking the turn when the claim cites no user-observable evidence.
Install
Transcribed from the repository README; not yet executed from a clean environment:
git clone https://github.com/bulletproofsoftware-ai/bulletproof-validation-gate.git cd bulletproof-validation-gate ollama pull llama3.2:3b # any Ollama model works; larger judges better ./install.sh # prints the settings.json snippet, paths filled in # paste the printed "Stop" hook block into ~/.claude/settings.json
bulletproof-validation-gate/docs/media/1Problem Statement
AI coding assistants routinely declare success without proof. The claim "fixed it" arrives with no test output, no rendered page, no command result — and the burden of discovering that nothing was fixed shifts onto the human. Every unverified claim the user has to disprove themselves erodes trust in the assistant.
The conventional mitigations are weak. Asking the model to self-check produces confirmation, not scrutiny: a model that just claimed success is poorly positioned to judge that claim. CI gates catch broken code but run too late and say nothing about whether the assistant's *description* of its work was honest.
This gate moves the check to the moment the claim is made, and makes the verifier adversarial. A second model — running locally, with no stake in the first model's output — is prompted to refute the claim. If it cannot find cited, observable evidence, the turn is blocked and the assistant must substantiate or retract.
2Architecture
The entire tool is a single script, completion_claim_gate.py, invoked by Claude Code on the Stop event. It reads the hook payload from stdin — last_assistant_message, session_id, transcript_path, stop_hook_active — and decides whether to emit a pass-through or a block.
Regex pre-filter
COMPLETION_RE matches 14 patterns (\bdone\b, \bfixed\b, \bdeployed\b, \bshould (?:resolve|work|fix)\b, and similar). No match means no LLM call — emit_pass_through() prints {} and the turn proceeds. This keeps the common case free.
Evidence gathering
read_transcript_tail() reads the last ~40 JSONL transcript entries, truncating each to 1000 characters and capping the whole context at 8000, so the verifier judges the claim against what actually happened in the session.
Adversarial verifier
call_verifier() POSTs to Ollama /api/generate with temperature: 0.1 and num_ctx: 8192. The prompt names three PASS conditions and four FAIL conditions, and instructs the model to disprove rather than confirm.
Defensive parsing
The response parser strips <think> preambles and code fences before json.loads, expecting {"verdict", "reason", "missing_evidence"} and adding latency_ms. Malformed output is treated as UNPARSEABLE, not as a failure.
Fail-open by design
emit_block() prints {"decision":"block", ...} on FAIL only. PASS, ERROR, and UNPARSEABLE all pass through, so an unreachable or slow Ollama can never wedge a session.
Safety Properties
- Loop protection — if
stop_hook_activeis set, meaning the turn was already triggered by a prior block, the gate passes through unconditionally and cannot trap the session in a block cycle. - Bounded cost — the regex stage rejects the overwhelming majority of turns before any model is invoked; only claim-shaped messages reach the verifier.
- Bounded latency —
OLLAMA_VERIFIER_TIMEOUT_S(default 20s) caps the verifier, and the hook's owntimeoutof 35000ms must exceed it. - Audit trail — every invocation writes a verdict file named
<UTC-timestamp>_<session-id[:8]>.jsoncontaining a 2000-character claim excerpt plus the verdict object, and appends to a log. Both writes swallow exceptions so audit failure never breaks the turn. - Safe notification —
fire_notify()shells out viasubprocess.runwithshell=False, an argv list, and a 5s timeout, avoiding shell injection from claim text.
3Requirements
Requirements are derived from the implemented behaviour of completion_claim_gate.py.
| ID | Requirement |
|---|---|
| REQ-VG-001 | Intercept the Claude Code Stop event and read the hook payload from stdin. |
| REQ-VG-002 | Apply a regex pre-filter of 14 completion patterns before invoking any model. |
| REQ-VG-003 | Pass through immediately when no completion language is present. |
| REQ-VG-004 | Read the transcript tail as bounded evidence context (≈40 entries, 8000 chars). |
| REQ-VG-005 | Prompt the verifier adversarially — to disprove the claim, not confirm it. |
| REQ-VG-006 | Parse verifier output defensively, stripping think-tags and code fences. |
| REQ-VG-007 | Block only on an explicit FAIL verdict; pass through on ERROR or UNPARSEABLE. |
| REQ-VG-008 | Pass through unconditionally when stop_hook_active is set. |
| REQ-VG-009 | Write a per-invocation JSON verdict file and append to an audit log. |
| REQ-VG-010 | Never let audit or notification failure affect the gate decision. |
| REQ-VG-011 | Operate with zero third-party Python dependencies. |
4Interfaces
The gate has no HTTP API, no CLI subcommands, and no MCP tools. It is configured entirely through the Claude Code hook definition and environment variables.
| Variable | Default | Purpose |
|---|---|---|
| OLLAMA_URL | http://localhost:11434/api/generate | Verifier endpoint |
| OLLAMA_VERIFIER_MODEL | llama3.2:3b | Model used to judge the claim |
| OLLAMA_VERIFIER_TIMEOUT_S | 20 | Verifier timeout in seconds |
| OLLAMA_VERIFIER_KEEP_ALIVE | 24h | Ollama model keep-alive window |
| VALIDATION_GATE_VERDICT_DIR | (unset) | Directory for per-invocation verdict files |
| VALIDATION_GATE_LOG | (unset) | Append-only audit log path |
| VALIDATION_GATE_NOTIFY | (unset) | Executable invoked on a FAIL verdict |
5Integration Points
The gate deliberately has a very small integration surface — a local model and the hook system, nothing else.
- Claude Code — registered as a
Stophook runningpython3 <abs path>/completion_claim_gate.pywithtimeout: 35000. - Ollama — the local verifier backend, reached over
/api/generate. No cloud model is contacted and no claim text leaves the machine. - Operator notification — an optional executable named by
VALIDATION_GATE_NOTIFY, invoked only on FAIL. - No database, no Qdrant, no Postgres, no n8n. State is limited to verdict files and a log.
6Repository
The implementation is a single script; the remainder of the repository is documentation, install tooling, and assurance evidence.
| Path | Purpose |
|---|---|
| completion_claim_gate.py | The entire implementation — filter, verifier call, parsing, audit |
| install.sh | Prints the settings snippet with the absolute path resolved |
| settings.snippet.json | Hook definition to paste into Claude Code settings |
| tests/test_gate.sh | Three smoke tests |
| docs/ | OVERVIEW, INSTALL, HOW-TO-USE, ADMINISTRATOR, SBOM |
| docs/scan/ | SARIF output, attestation, and scan reports |
| .github/workflows/ci.yml | Python 3.12 CI with SHA-pinned actions |
7Implementation Notes
Operational characteristics worth knowing before deployment.
- The hook
timeoutmust exceedOLLAMA_VERIFIER_TIMEOUT_S, or Claude Code will kill the gate before the verifier answers. - The hook references the script by absolute path — moving the file breaks the hook until the path is updated.
- Small 3B models err toward blocking; larger verifier models judge claims more accurately at higher latency.
- The gate only blocks when Ollama is reachable. With the backend down it is inert by design.
- It judges the *claim*, not the code. It is explicitly not a linter, test runner, or CI gate.
install.shprints the settings snippet — it does not edit Claude Code settings for you.