Exploit for llm-redteam-helixpay

## https://sploitus.com/exploit?id=3DD9F1CA-8B7C-5E32-9E5F-A3E794ECC46C
# Red-teaming an LLM support agent — HelixPay

A self-contained AI security project: a **deliberately vulnerable LLM application**, a
**red-team harness** that attacks it, and a **hardened rebuild** that defeats every
attack — with reproducible before/after numbers.

> **Result:** 11 catalogued attacks, mapped to the OWASP LLM Top 10.
> **11/11 succeed** against the vulnerable app. **0/11 succeed** against the hardened app.
> Runs fully offline with no API key; the same harness runs against the live Claude API.

🔗 **Live showcase:** https://gfelobes.github.io/llm-redteam-helixpay/

📄 **The write-up is [`REPORT.md`](REPORT.md)** — threat model, per-finding PoCs, root
causes, and mitigations. This README is the quickstart.

---

## The target: "HelixPay"

`app/` is a fictional fintech support assistant — an LLM with a system prompt, a RAG
knowledge base, and three tools (`get_account_balance`, `issue_refund`,
`reset_password`). It ships in two forms:

| File | What it is |
|------|------------|
| [`app/vulnerable_app.py`](app/vulnerable_app.py) | The naive "wire the LLM to the tools" build. Secret in the prompt, retrieved content spliced into context, tool calls executed verbatim, output returned raw. |
| [`app/hardened_app.py`](app/hardened_app.py) | Same model, same tools — wrapped in the controls from [`app/defenses.py`](app/defenses.py): input guardrail, spotlighting, **code-enforced tool authorization**, and output sanitization. |

The one file to read if you read nothing else is [`app/defenses.py`](app/defenses.py) —
the mitigations, as real code.

## Quickstart

```bash
# No dependencies, no API key — the default backend is a deterministic offline
# model that reproduces documented LLM failure modes.
python -m redteam.runner        # runs 11 probes against both apps, writes results/
python -m unittest              # 12 tests proving the mitigations hold
```

Play with it:

```bash
python scripts/demo.py --agent vulnerable --attack P07   # watch an attack land
python scripts/demo.py --agent hardened   --attack P07   # watch it get blocked
python scripts/demo.py --agent hardened                  # interactive chat
```

### Run it against a real model

Every attack and defense also runs against the live Claude API — no code changes:

```bash
pip install anthropic
export ANTHROPIC_API_KEY=sk-...
HELIX_BACKEND=live python -m redteam.runner
```

## How the offline backend is legitimate

The vulnerabilities demonstrated here live in the **application** — missing tool
authorization, secrets in context, untrusted content treated as instructions,
unescaped output — **not** in a model's weights. The offline backend
([`app/model.py`](app/model.py)) is a scripted stand-in that reliably reproduces the
behavior an attacker coaxes out of a real LLM (following injected instructions,
leaking its prompt, over-calling tools). That lets any reviewer reproduce every
finding bit-for-bit with no key and no network — and it lets the harness prove the
**defenses work regardless of which model sits behind them**. Flip `HELIX_BACKEND=live`
to confirm the same controls hold against a real model.

## Layout

```
app/
  model.py           # backend: offline mock (default) + live Claude API
  vulnerable_app.py  # the target — intentionally insecure
  hardened_app.py    # the fix — same model, wrapped in controls
  defenses.py        # ★ the mitigations, as reusable code
  tools.py           # tools + fake datastore + side-effect ledger
  knowledge_base.py  # RAG docs incl. one poisoned support ticket
redteam/
  probes.py          # the attack catalog (11 probes, OWASP-mapped)
  scorer.py          # impact-based success detection
  runner.py          # runs probes vs both apps → results/
  report.py          # renders the comparison table
tests/               # unittest suite (no third-party deps)
scripts/demo.py      # interactive / one-shot CLI
results/             # generated findings.json + results.md (checked in)
REPORT.md            # ★ the red-team report
```

## Scope & ethics

This is a defensive-security teaching artifact against a fictional app the author
controls. The "attacks" are prompt-injection and access-control test cases; the point
is the mitigations. Don't point the harness at systems you're not authorized to test.