## https://sploitus.com/exploit?id=B59AAC7A-707D-5D28-9000-839793E1D991
# secreport-bench โ the Security-Report Benchmark
An evaluation harness that measures the **usefulness** of AI-generated security
reports.
Existing vuln benchmarks ask *"can an AI fix a bug it can see?"* โ they hand the
fixer the full code. This one asks a different question:
> **Is the AI's report good enough that a _different_ agent can kill the bug
> using _only the report_?**
An **Auditor** model reads vulnerable code and writes a security report. We then
**hide** the auditor's reasoning, the bug's exact location, and the gold fix. A
fixed **Fixer** model receives *only the report* (plus a copy of the code, with
the vulnerable region blanked) and must patch the bug. We then run the
**exploit**: if it no longer fires, the report was useful; if it still fires,
the report was weak โ no matter how professional it read.
**The exploit is the oracle.** No human judge, no LLM-as-judge.
This is defensive evaluation infrastructure in the lineage of SWE-bench,
SEC-bench, and Round-Trip Correctness: it runs *known* exploits against *known*
vulnerable code purely to verify whether a patch holds.
---
## Names
*RELAY* is the benchmark/method; **`secbench`** is the Python package and CLI
that implements it; *secreport-bench* is this repository. All three name the
same project.
---
## Quickstart (zero API keys)
```bash
pip install -e .
secbench validate ./tasks # static task-spec checks -> "all tasks valid"
secbench selftest ./tasks # executable oracle gate: unpatched fires, gold fix -> FIXED
secbench run --mode all # run all 9 tasks in all 3 modes with the mock model
secbench score ./results/*.json # (re)build the leaderboard
```
`leaderboard.html` shows RUS per mode and the report-lift number. A run also
prints a summary to stdout, e.g.:
```
mode RUS
------------------------------
no-report 0% (0/1)
with-report 100% (1/1)
gold-answer 100% (1/1)
report lift (with-report โ no-report): +100%
```
The default `mock` auditor/fixer is deterministic and offline: its fixer only
produces an effective fix when it is handed a report, so a fresh clone
demonstrates report lift end-to-end without any network calls.
---
## The three modes
Same fixer, same exploit โ only the fixer's input changes:
| mode | fixer sees | role |
|------|------------|------|
| `no-report` | code only (bug region blanked) | baseline |
| `with-report` | code (blanked) + the auditor's report | **the thing we test** |
| `gold-answer` | code (blanked) + the gold reference finding | best-case ceiling |
**Report lift** = `RUS(with-report) โ RUS(no-report)` is the headline signal.
```bash
secbench run --mode with-report # just the tested condition
secbench run --mode all # all three, diffed on the leaderboard
secbench run --auditor mock --fixer mock # explicit (these are the defaults)
```
---
## Scoring
- **Report Usefulness Score (RUS):** fraction of tasks where, in a given mode,
the fixer's patch makes the exploit stop firing (outcome `FIXED`). Every other
outcome โ still firing, patch failed, timeout, error โ counts against RUS.
- Aggregated per mode; **report lift** is displayed alongside.
- Outputs land in `results/`: `run-.json`, `run-.csv`, and
a single static `leaderboard.html` (no web server).
- Hooks are stubbed (not yet implemented) for **completeness gap** (a held-out
sibling exploit still fires) and **hallucination rate** (a report asserting a
bug no exploit can verify).
---
## The sandbox (safety-sensitive)
Each evaluation runs in a **fresh, throwaway workspace**: the vulnerable tree is
copied in, the fixer's patch is applied, the exploit runs, the signal is
matched, and everything is torn down. Two backends implement the same interface
and are selected automatically:
- **`docker` (default when available):** runs the exploit inside a fresh
container from `Dockerfile.base` with `--network none`, memory/pid limits, and
`--rm`. Exploit code never touches the host. This is the real isolation.
- **`local` (fallback):** a development backend used when Docker is absent. It
runs the exploit in a temp-dir workspace as a subprocess with best-effort
resource limits and a hard timeout. It **refuses any task not marked
`host_safe: true`**. Force it with `--force-local`.
> For untrusted or unsafe tasks, install Docker โ the container backend is then
> chosen automatically. The bundled `example-cmdi` task is `host_safe` (no
> network, no destructive actions) and runs under either backend.
Every failure mode is a distinct, recorded outcome, never a crash:
`FIXED`, `FIXED_BUT_BROKEN`, `NOT_FIXED`, `PATCH_FAILED`, `EXPLOIT_TIMEOUT`, `ERROR`.
A task may declare a `functional_check`; a patch that stops the exploit but
breaks the app's legitimate behaviour is scored `FIXED_BUT_BROKEN` (a shallow
fix), not `FIXED`.
---
## Self-test & CI
`secbench selftest ./tasks` executes each task's oracle end-to-end: the
unpatched code must fire the exploit, and the gold fix must reach `FIXED`
(exploit stopped **and** the functional check passes). `secbench run` gates on
this and skips any task whose oracle is unsound in the current environment.
GitHub Actions (`.github/workflows/ci.yml`) runs the unit tests and the
self-test on every push.
Leakage control: the fixer sees the vulnerable region blanked with a marker
**plus decoy blanks elsewhere**, so the marker never pinpoints the exact bug
location.
---
## Task format (`task.yaml`)
```yaml
id: example-cmdi
title: Command injection in filename handler
language: python
vuln_class: command-injection
entrypoint: app.py
vulnerable_path: vulnerable/ # code given to the fixer
bug_file: app.py # the file that contains the flaw
bug_hint_region: [13, 16] # 1-based line range blanked for leakage control
host_safe: true # may run under the local sandbox fallback
exploit:
cmd: "python3 exploit/poc.py" # runs inside the sandbox, cwd = workspace root
fired_signal: "PWNED" # substring meaning "exploit succeeded"
timeout: 30
gold:
patch: gold/fix.patch # reference patch (self-test + optional scoring)
finding: gold/finding.md # reference report (used in gold-answer mode)
```
A task directory bundles `task.yaml`, `vulnerable/`, `exploit/`, and `gold/`.
The `gold/` tree is never copied into the sandbox.
### Adding a task
1. Create `tasks//` with the four pieces above.
2. Make the exploit emit `fired_signal` **only** when the bug actually triggers
(prefer a side effect like a marker file over string-matching program output,
so a patched app that echoes the payload back can't cause a false positive).
3. `secbench validate ./tasks` โ then confirm the ground truth:
the exploit fires unpatched and stops after `gold/fix.patch` applies.
---
## Fixer output contract
The fixer returns either the **complete corrected file** (primary, most robust)
or a **unified diff**; the runner auto-detects and applies it (`patch -p1`/`-p0`
for diffs, with headers synthesized for a bare hunk). The mock returns full
files; the Anthropic adapter is prompted to do the same.
Because the fixer's output is untrusted, diff application is hardened: any diff
whose target path is absolute or contains `..` is refused (`PATCH_FAILED`) so a
malicious diff cannot write outside the workspace. Both sandbox backends
interpret `exploit.cmd` through a shell (`sh -c`), so the exploit oracle is
backend-independent.
---
## Real models
```bash
pip install -e ".[anthropic]"
export ANTHROPIC_API_KEY=sk-ant-...
secbench run --mode all --auditor anthropic:claude-opus-4-8 --fixer anthropic:claude-sonnet-5
```
Any provider plugs in behind the `Model` interface (`secbench/models/base.py`):
```python
class Model(Protocol):
name: str
def audit(self, code: str, task_meta: dict) -> str: ... # -> security report
def fix(self, prompt: str) -> str: ... # -> corrected file or diff
```
---
## Layout
```
secbench/ engine (importable library)
cli.py CLI: run | selftest | score | validate
config.py run config, modes, constants
models/ base interface, mock, anthropic
pipeline/ audit -> redact -> fix ; the three modes
sandbox/ backends (docker/local) + runner + Dockerfile.base
scoring/ metrics (RUS, lift) + JSON/CSV/HTML report
tasks/loader.py task spec loading + validation
tasks/ the dataset (kept separate from engine code); 9 tasks:
command-injection, path-traversal (x2), SQLi, insecure
deserialization, YAML load, tar-slip, mass-assignment,
format-string -- each with vulnerable/, exploit/, gold/
results/ run outputs (gitignored)
tests/ pytest: validation, mode routing, oracle, functional check
.github/workflows/ CI: pytest + selftest on every push
```
## Development
```bash
pip install -e ".[dev]"
pytest
```
## Scope (MVP)
Deliberately **not** built: web service / hosted leaderboard, auth, database
(flat files only), parallel/distributed execution, plugin system, multi-language
support beyond Python. The effort is concentrated on the sandbox and the
three-mode experiment.