## https://sploitus.com/exploit?id=F9E3D8FD-FB87-50DD-8B68-34372B55894A
# Starter task: designing tasks Fable (Claude Code) and GPT-5.6 Sol (Codex) can't do
Prof. Daniel Kang (UIUC) asked for a starter task before a research
discussion: design a task specification, create 2-3 tasks, validate that
Fable (Claude Code) and GPT-5.6 Sol (Codex) genuinely cannot solve them, and
write a short report. This repo is the full, real work behind that โ
including the parts that didn't pan out.
**Start here: [`REPORT.md`](REPORT.md) / [`REPORT.pdf`](REPORT.pdf)** โ the
actual submission. **[`VALIDATION_STATUS.md`](VALIDATION_STATUS.md)** has
the complete real evidence trail for every task attempted, including the
ones abandoned along the way.
## What's in here
- **`task_05_cve_exploit/`** โ the final, submitted task. Two real,
live-verified CVE exploitation challenges built on Prof. Kang's own
open-source benchmark, [CVE-Bench](https://github.com/uiuc-kang-lab/cve-bench).
GPT-5.6 Sol was run against both (Fable's CLI auth was broken all
session โ see below). Real finding: Sol didn't fail from a skill
ceiling, it got blocked mid-task by OpenAI's own safety classifier, both
times, across two unrelated CVEs and two different prompt designs.
- **`task_04_sparse_blackbox_attack/`** โ an earlier, abandoned candidate:
a query-efficient sparse black-box adversarial attack task against a
real RobustBench-published adversarially-trained model. Real
infrastructure, real calibration numbers, but paused mid-calibration
(heavy local compute, see `VALIDATION_STATUS.md`) in favor of Task 5.
- **`task_01_agent_benchmark_audit/`, `task_02_swebench_temporal_inconsistency/`,
`task_03_swebench_cost_consistency/`** โ the first three candidates, all
built on real SWE-bench Verified data. All three were disqualified:
two were genuinely solved by GPT-5.6 Sol, and the third turned out to
have no real ground truth at all (caught by exhaustively re-deriving the
"correct" answer against live data โ see `VALIDATION_STATUS.md`).
- **`task_runs/`** โ raw transcripts and isolated work copies from every
real agent run referenced in the report.
## Why this is public
Because the finding is itself about agent evaluation and AI safety
behavior โ the kind of thing worth being checkable, not just asserted.
## Note on Fable
Claude Code's CLI returned HTTP 401 on every inference call for the entire
session, including after a completed OAuth re-login. This looks like an
account-side issue rather than something fixable client-side, so the final
submission is validated against GPT-5.6 Sol only.