## https://sploitus.com/exploit?id=381D5D17-B033-5BF1-9D4B-86CB6311DBB7
# waf-poc โ Choreo CP WAF Bake-Off (OpenResty)
Three-way WAF evaluation for the post-migration Choreo control plane on OpenResty. See `Plan.md` for full context, `tasks.md` for the breakdown, `summary-comparison.md` for the engine comparison memo, and `CLAUDE.md` for the cold-start brief.
## What this is
Sibling images, all `FROM openresty/openresty:1.29.2.3-alpine-fat`, each carrying one WAF engine. Same nginx.conf, same vhosts, same CRS โ engine is the only variable.
| Track | Engine | Status |
|---|---|---|
| A (primary) | ModSecurity v3 + ModSecurity-nginx + OWASP CRS 4.x | M1 cycle complete (local Docker). |
| B | Coraza + coraza-nginx + same CRS 4.x | Not started. |
| C | open-appsec attachment (ML) | Not started. |
The current cycle (`plan-m1.md`) narrowed to **ModSec only** to prove the OpenResty + libmodsecurity3 + ModSecurity-nginx + CRS combo works end-to-end. Tracks B/C are scaffolded but empty.
Pinned versions (Track A): libmodsecurity3 `v3.0.15`, ModSecurity-nginx `v1.0.4`, OWASP CRS `v4.26.0`.
## Layout
```
docker/{a-modsec,b-coraza,c-openappsec}/ per-engine Dockerfiles (only a-modsec is populated)
nginx/ eval-cycle nginx.conf (DP vhost porting deferred)
modsec/ coraza/ openappsec/ per-engine config + exclusions
crs/ OWASP CRS 4.x (git submodule)
k8s/{a-modsec,b-coraza,c-openappsec}/ per-engine k8s manifests (empty)
tests/smoke/ curated cases.tsv (30 attacks + 10 FPs), per-rule diagnostics
tests/gotestwaf/ Wallarm gotestwaf (~675 attacks, 141 benign), breadth check
bakeoff/ matrix, FP tally, decision memo (Phase-1.5 artifacts)
```
## Build
```
make build-a # ModSec image (~16 min on a clean cache, seconds with cache)
```
Verify the image is healthy:
```
make nginx-test # expects: rules loaded 0/845/0, libmodsecurity3 v3.0.15
```
If you see fewer than 844 rules, the CRS submodule isn't initialized โ `git submodule update --init crs`.
## Running tests
Two independent harnesses by design. **Smoke** is the primary signal (per-rule traceability via `X-Eval-Case-ID`). **gotestwaf** is the breadth check (bundled OWASP/community corpus).
### Knobs
Both harnesses read these env vars:
| Env var | Values | Default | Effect |
|---|---|---|---|
| `MODSEC_MODE` | `detect`, `on`, `off` | `detect` (smoke runs both) | `SecRuleEngine` setting. `detect` = log only; `on` = enforce (403 on anomaly threshold); `off` = engine disabled. |
| `CRS_PARANOIA` | `1`, `2`, `3`, `4` | `1` | CRS Paranoia Level. PL1 = baseline. PL2+ enables progressively stricter rule sets at the cost of more FPs. |
| `PORT` | any free port | `8080` (smoke), `8081` (gotestwaf) | Host port for the WAF container. Override if the default is in use (e.g., k3d on 8080). |
| `IMAGE` | image tag | `waf-poc-modsec:latest` | WAF image to test. |
The same image runs all combinations โ the entrypoint templates `/etc/modsec/mode.conf` (mode) and `/etc/modsec/paranoia.conf` (PL) at container startup. No rebuild needed to switch mode or PL.
### Smoke (`tests/smoke/`)
Curated `cases.tsv` (30 attacks + 10 FPs). Runs both `detect` and `on` modes back-to-back, joins audit logs against cases via `X-Eval-Case-ID`, emits `matrix.md`.
```
# Default: PL1, runs detect + on, matrix.md emitted.
make eval-m1
# PL2.
CRS_PARANOIA=2 bash tests/smoke/run-eval.sh
# Different host port (e.g., when k3d holds 8080).
PORT=8090 bash tests/smoke/run-eval.sh
# Combined.
CRS_PARANOIA=2 PORT=8090 bash tests/smoke/run-eval.sh
```
Output: `tests/smoke/results/matrix.md` (and per-mode `*.cases.tsv`, `*.audit.jsonl`, `*.container.log`).
### gotestwaf (`tests/gotestwaf/`)
Wallarm gotestwaf via the official `wallarm/gotestwaf:latest` Docker image. Reports filenames are tagged `gotestwaf--pl.{html,json,csv}` so multiple runs accumulate in `tests/gotestwaf/results/`.
```
# Default: enforce mode, PL1.
make eval-gotestwaf
# Detect mode, PL1 (informational โ gotestwaf scores blocks, so detect runs show ~0% TP by design).
make eval-gotestwaf-detect
# Enforce mode, PL2.
CRS_PARANOIA=2 bash tests/gotestwaf/run.sh
# Override port and image.
PORT=8085 IMAGE=waf-poc-modsec:dev bash tests/gotestwaf/run.sh
```
Open the HTML report:
```
open tests/gotestwaf/results/gotestwaf-on-pl1.html
```
### Manual run
If you want to poke at the container directly without the harnesses:
```
docker run --rm -e MODSEC_MODE=on -e CRS_PARANOIA=2 -p 9090:8080 waf-poc-modsec:latest
# In another shell:
curl -i -H "Host: apis.choreo.dev" "http://127.0.0.1:9090/?q=1' OR '1'='1"
```
The `Host: apis.choreo.dev` header is load-bearing โ without it CRS rule **920350** (numeric IP host) fires on every request because we hit `127.0.0.1`.
## Results so far (Track A)
### M1 baseline at PL1
- **Smoke:** 28/30 attacks detect-then-block; 10/10 FPs clean. The 2 misses (A080/A081) are pre-ModSec nginx 400s โ defense-in-depth, not detection failures.
- **gotestwaf:** overall 63.20%, TP 48.19% (320/664), TN 90.51% (124/137 benign passed), 13 FPs.
### PL2 (single-knob bump)
| | PL1 | PL2 |
|---|---|---|
| Smoke attacks blocked | 28/30 | 28/30 |
| Smoke FPs clean | 10/10 | **9/10** (F007 โ CRS 942380) |
| gotestwaf overall score | 63.20% | **71.98%** |
| gotestwaf TP | 48.19% | **54.75%** |
| gotestwaf TN | 90.51% | **62.41%** |
| gotestwaf FPs | 13 | **53** |
Headline: PL2 gains ~9 points overall (mostly on encoder/placement evasions in LDAP, mail, shell, NoSQL, and RCE-in-URL test sets) but quadruples FPs on benign English text. Default stays at PL1; PL2 promotion is gated on a curated exclusion bundle. See the comparison output in `tmp/` and the M1 matrix in `tests/smoke/results/matrix.md`.
## Bake-off artifacts (broader plan)
Phase 1 fills these in when scope re-widens beyond M1:
- `bakeoff/matrix-template.md` โ per-attack ร per-engine outcome
- `bakeoff/fp-tally-template.md` โ per-engine FP tally and per-rule decisions
- `bakeoff/decision-memo-template.md` โ Phase-1.5 gate sign-off (which engine enforces, which stay detect-only, which are removed)
The decision memo is a hard deliverable. Without it the bake-off devolves into "ModSec works, ship it" and wastes the Coraza/open-appsec investment.
## Status
M1 (ModSec eval cycle, local Docker) โ **complete**. Phase 1 (k8s deploy + real CP traffic) โ not started. See `tasks.md` and `CLAUDE.md` for current state.