## https://sploitus.com/exploit?id=52206818-253F-5857-8232-0B06A258879D
# ExploitGym for Inspect
This repository runs the pinned ExploitGym corpus under Inspect while retaining
the repository's native Codex, Claude Code, or Gemini CLI agent loop. The CLI
still decides when to call the model, runs its own local tools, and writes its
native trajectory. Inspect owns model inference, sample orchestration,
transcripts, scoring, and teardown.
```text
native CLI -> ExploitGym route/retrieval/model/budget policy
-> private Inspect bridge -> configured Inspect model
```
This is a maximum-parity implementation, not a claim of completed parity
certification. A real one-sample end-to-end run for each supported CLI and task
family still has to pass on a designated Linux host before release. See the
[production parity guide](docs/parity.md) for the contract, accepted drift, and
certification checklist.
Inspect `0.3.260` does not natively expose the Anthropic or Google token-count
routes, so this integration supplies a separate count-only bridge. It resolves
only predeclared exact model aliases and invokes the matching evaluator-side
provider SDK, keeping provider credentials out of the sandbox and introducing
no local token estimate. Claude/Gemini still require a real-provider comparison
before certification because provider headers, retry behavior, and SDK response
normalization are not byte-identical to a direct CLI connection.
## Pinned evaluation surface
- ExploitGym: `e4123d043774623b2274e6bbe0155a423d631f0a`
- Inspect AI: `0.3.260`
- Codex CLI `0.120.0` with `gpt-5.3-codex`
- Claude Code `2.1.119` with `claude-sonnet-4-6`
- Gemini CLI `0.37.2` with `gemini-3.1-pro-preview`
- 869 unique tasks: 186 kernel, 502 user, and 181 V8
The default `subset=sample` is the pinned upstream 20-task sample. Use
`subset=all` for the full catalog.
## Supported host and installation layout
Real evaluation requires a native Linux x86_64 Docker host. Kernel parity also
requires accessible `/dev/kvm`; disabling that check permits uncertified
software emulation only. macOS is supported for task discovery and safe tests,
not for real ExploitGym execution.
ExploitGym's standard no-mitigation tuple is host ASLR disabled
(`kernel.randomize_va_space=0`), user `exp.none`, V8 `nodefense`, and kernel
`default`. Upstream `pre_run.py --hardened` selects host ASLR enabled, user
`exp.hardened`, V8 `strict`, and kernel `strict`. Inspect exposes these as the
same independent native-compatible task arguments, so any other combination is
a custom run. The task checks and records the declared `host_aslr` state but
never changes this host-wide setting. It also requires the declared host
core-dump filename pattern, defaulting to ExploitGym's recommended
`core.%e.%p.%t`, rather than allowing a host pipe handler to consume agent core
dumps. Disabling ASLR weakens every host and container process, so standard runs
belong only on a dedicated disposable machine; restore both host settings
afterward.
Run from a source checkout at the repository root. An editable install is fine,
but a wheel or site-packages-only installation is not a supported full-run
layout: the production Docker build needs both this source tree and the
independently locked `sidecars/native_proxy` project. Docker Compose 2.21 or
newer is required for the kernel Compose file's `extends` support. Docker
Engine 28 or newer is required so every private bridge can use isolated gateway
mode rather than exposing host services to agent containers.
The exact safe-test gate also requires `uv` 0.11.28 and Node 22.21.0 on
`PATH`. Node is a host-side test dependency here; production samples use the
separately built, fully static runtime described below.
```bash
uv sync --frozen
```
Inspect's Compose initialization builds services that are missing or out of
date. The runtime build compiles the repository's static Node and pinned CLI
bundle. Claude Code's pinned musl payload is relocated to a licensed loader in
the trusted `/data` tree so it also launches in the Ubuntu task corpus. The
first run can be substantial. This optional warm-up uses the same commit-scoped
tag as the production Compose file:
```bash
docker buildx build \
--platform linux/amd64 \
--file src/exploitgym_eval/Dockerfile.runtime \
--tag exploitgym-inspect-runtime:0.1.0-e4123d043774 \
--load \
.
```
The task validates the resulting image's platform, upstream revision, static
Node version, and CLI-version labels before starting the agent.
After the native install script and non-secret CLI configuration writes, setup
restarts the exact already-running agent container before creating the
token-bearing workspace or provisioning any capability. It verifies the same
container ID, observed image ID, install-only network, running state, private
PID namespace, and Docker-injected init configuration. The static deadline
helper later verifies the live Docker init as PID 1 and `/data/idle` as its
unique trusted direct child before releasing the CLI. This clears
installer-launched background processes before any task secret exists.
The Docker-socket controller exposes only bounded lifecycle operations;
`/run_command` is disabled. A filesystem-backed admission gate covers create,
restart, health, and delete calls, and teardown seals new untrusted operations
and drains every admitted operation while retaining an authenticated cleanup
delete path. Before challenge code starts, the controller verifies that the
dynamic target uses the exact setup-observed agent image ID, exactly one target
network, only the expected read-only mounts and devices, the configured
resource limits, no privileged mode or extra capabilities, and no published
ports. The observed target image ID and available repository digests are
recorded as provenance.
Native CLI deadlines use the read-only fully static `/data/deadline` helper.
Its one-purpose capability is consumed by the helper parent and is not passed
to the CLI. In production container mode it first proves the private Docker
init/idle topology. At the monotonic deadline it freezes every other process in
the agent PID namespace through pidfds and stable fixed-point `/proc` scans,
asks policy to durably close admission, cancel active inference, and snapshot
request sequences under a fresh marker ID, and only then terminates the frozen
set and returns 124. `setsid()` and double-fork descendants cannot race the
marker or later flag collection. A marker, quiescence, cleanup, or persistence
failure is infrastructure status 125, never a benchmark timeout.
Ordinary clean and nonzero CLI exits deliberately retain native GNU-timeout
semantics: detached work is not reaped merely because the direct CLI returned.
A delayed exploit that writes the native flag before grading is therefore still
valid benchmark behavior. Cancellation, internal failure, and the helper's own
deadline instead contain the entire agent process namespace before returning.
Evaluator-owned Node helpers never inherit challenge-image execution hooks.
They run through a static read-only launcher that replaces the environment,
permits only `/data/node/bin/node`, and pins OpenSSL to immutable configuration
and disabled module/engine paths on `/data`. The native CLI remains the native
agent loop; its deadline parent only strips known loader/runtime hook variables.
## Safe validation
These commands discover and test the integration without launching an
autonomous exploit agent against a challenge target:
```bash
uv run inspect list tasks
uv run ruff check .
uv run pytest -m "not docker" -q
uv sync --project sidecars/native_proxy --frozen
uv run --project sidecars/native_proxy ruff check sidecars/native_proxy
uv run --project sidecars/native_proxy pytest sidecars/native_proxy/tests -q
```
With Docker available, the deterministic protocol probe uses a mock model and a
minimal client; it does not install or execute a native agent CLI:
```bash
uv run pytest tests/test_protocol_docker.py -q
```
The opt-in pinned Codex bridge test is Linux x86_64-only and still uses a mock
model rather than a real ExploitGym target:
```bash
RUN_CODEX_BRIDGE_TEST=1 \
uv run pytest tests/test_codex_docker.py -q -s
```
None of the macOS-safe or mock-model checks replaces a real Linux end-to-end
certification run.
### OpenRouter compatibility diagnostics
OpenRouter can be exercised without replacing the native Codex, Claude Code,
or Gemini CLI loop. Set `inference_backend=openrouter_diagnostic` and select the
matching model route below. This mode is deliberately tagged
`non-parity`: inference is normalized through Inspect and routed by OpenRouter,
and Claude/Gemini preflight token counts use Inspect's deterministic local
estimate because OpenRouter has no native count endpoint. The OpenRouter key
stays in the evaluator process; the agent receives only its sample-scoped
policy bearer. Use a key with a provider-side spending limit: ExploitGym's
native-alias budget is post-response accounting and is not a hard OpenRouter
charge cap.
Codex uses Inspect's built-in OpenAI provider against OpenRouter's canonical
Responses endpoint. The doubled `openai/openai/...` model name is intentional:
the first segment selects Inspect's OpenAI provider and the remaining
`openai/...` is OpenRouter's service model. This preserves the native Responses
stream, optional function schemas, and custom tools such as `apply_patch`
without a policy-sidecar codec. The task rejects the Chat-based
`openrouter/openai/...` route because Inspect 0.3.260 would drop those
Responses-only custom tools.
The pinned defaults are:
| Native CLI | Inspect model | Transport |
| --- | --- | --- |
| Codex | `openai/openai/gpt-5.3-codex` | OpenRouter Responses |
| Claude Code | `openrouter/anthropic/claude-sonnet-4.6` | OpenRouter Chat Completions |
| Gemini CLI | `openrouter/google/gemini-3.1-pro-preview` | OpenRouter Chat Completions |
For example, a bounded Codex diagnostic is:
```bash
# This is an OpenRouter key. Inspect reads it from OPENAI_API_KEY because the
# Codex diagnostic deliberately selects Inspect's OpenAI Responses provider.
export OPENAI_API_KEY=...
uv run inspect eval exploitgym_eval/exploitgym \
--model openai/openai/gpt-5.3-codex \
--model-base-url https://openrouter.ai/api/v1 \
-T agent_name=codex \
-T inference_backend=openrouter_diagnostic \
-T max_budget=1.0 \
-T timeout_seconds=300 \
-T family=user \
-T first_n=1 \
-T acknowledge_offensive_risk=true \
--max-samples 1
```
Use the corresponding model and `agent_name=claude_code` or
`agent_name=gemini_cli` for the other native loops; those routes use
`OPENROUTER_API_KEY`. These runs are useful for
interoperability and agent-capability diagnosis, but must not be included in
native-provider parity or benchmark-certification totals. Gemini's retired
`gemini-3-pro-preview` helper has no current OpenRouter endpoint, so diagnostic
runs explicitly map that helper to `google/gemini-3.1-pro-preview` and record
the drift.
## First real Linux sample
Configure credentials for the selected Inspect provider, use an authorized
isolated host, and explicitly acknowledge the offensive workload. This command
runs one user-family sample with the pinned Codex profile:
```bash
uv run inspect eval exploitgym_eval/exploitgym \
--model openai/gpt-5.3-codex \
-T agent_name=codex \
-T model=gpt-5.3-codex \
-T reasoning_effort=medium \
-T subset=sample \
-T family=user \
-T first_n=1 \
-T acknowledge_offensive_risk=true \
--max-samples 1
```
`first_n=1` limits the dataset to one sample. Inspect's `--max-samples 1` only
limits concurrency, so both are intentional for the first certification run.
The policy service separately admits at most four simultaneous authenticated
model requests to bound aggregate buffers. That is an agent-visible security
drift from native execution: certification must show its schema-v4
`integration_rejection_count` stayed zero for every measured sample. The same
sticky counter covers integration-only pre-provider body/policy bounds and
pre-ASGI listener/header rejection, including incomplete heads consumed by the
trusted deadline or finalizer; these failures cannot masquerade as a valid
agent nonsolve after an earlier successful request.
Commands for Claude Code and Gemini CLI, every task-selection parameter, and
the full build/certification procedure are in [docs/parity.md](docs/parity.md).
The bridge fails closed on any native auxiliary-model name that has not been
declared. The seven helper names pinned in the current Gemini CLI are registered
automatically with exact same-name Google routes; any additional helper needs
an explicit `model_routes='["native-name=inspect/provider-model"]'` entry and
separate certification. A route is accepted only when the resolved Inspect
model's name is the same native name, so policy and budget accounting cannot
silently change identity.
## Artifacts are sensitive
The Inspect `.eval` log can contain the token-bearing task prompt, normalized
model messages, native CLI output, exploit code, and other data created by the
agent. The integration redacts the known policy bearer and never directly
exports the controller's expected-flag state. A successful agent can still put
the recovered flag in its own workspace, log, or model traffic, so this is not
a general-purpose secret scrubber.
Export the transcript-carried compatibility artifacts only into controlled
storage:
```bash
uv run exploitgym-export path/to/run.eval artifacts/
```
Use `--sample-id`, `--epoch`, and `--overwrite` when needed. Treat both the
source `.eval` log and exported files as sensitive offensive-security material.