recon2exploit

## https://sploitus.com/exploit?id=D9F5767A-8B3F-5196-BBCE-5F05C959C461
# recon2exploit

**recon2exploit** is a single-file security assessment orchestrator (`recon.py`) that runs an **ordered, eight-phase pipeline** on one or more domains. It combines passive and active reconnaissance, service fingerprinting, directory discovery, vulnerability research (NVD), optional exploitation hooks, and structured reporting.

Use it **only** on systems and domains you are **explicitly authorized** to test. For normal scans, **`--authorized-only`** is required (use **`--doctor`** or **`--diff-reports`** without it). It does not replace contracts, scope rules, or law.

---

## Table of contents

1. [What the tool does](#what-the-tool-does)
2. [End-to-end pipeline (8 steps)](#end-to-end-pipeline-8-steps)
3. [Architecture overview](#architecture-overview)
4. [Prerequisites](#prerequisites)
5. [Installation](#installation)
6. [Configuration (`.env`)](#configuration-env)
7. [Domains file](#domains-file)
8. [Usage and CLI](#usage-and-cli)
9. [External tools and PATH](#external-tools-and-path)
10. [Reports](#reports)
11. [Safety, scope, and ops flags](#safety-scope-and-ops-flags)
12. [Development and CI](#development-and-ci)
13. [Troubleshooting](#troubleshooting)
14. [Legal use](#legal-use)

---

## What the tool does

| Area | Behavior |
|------|----------|
| **Asset discovery** | Subdomains (**crt.sh** with retries/backoff, optional **Certspotter** / **SecurityTrails** when API keys are set, **subfinder**), DNS resolution to IPs, heuristic **email** discovery (MX patterns, WHOIS with retries, light HTTP scrape). |
| **Fingerprinting** | Async TCP probes on common ports, optional **nmap** NSE passes for deeper service/version data. |
| **Web recon** | **gobuster** directory scan (orchestrator step), **priority path** HEAD/GET checks, integrated **recon_domain** (DNS, headers, per-IP nmap, optional gobuster, content-derived paths, verification). |
| **Attack surface** | Full DNS (A/AAAA/MX/NS/TXT/CNAME/SOA, SPF/DMARC), **WAF** signatures (12+), **subdomain takeover** (CNAME chain + HTTP fingerprints, 40+ services), **robots.txt / sitemap** sensitive-path scoring, **CORS** severity, **TLS/SSL** hints, **open redirect** probing with canary confirmation, per-target **risk score 0–100**. |
| **Web vulns** | Async **SQLi** (error + boolean-blind + time-based), context-verified reflected **XSS**, **LFI / path traversal**, **SSRF** cloud-metadata + internal IP probes, exposed **`.git`/`.env`/`actuator`/swagger** files, HTTP **security headers audit**, and per-subdomain replay for up to 25 live hosts. |
| **Entry points** | BFS crawler extracts every **URL parameter**, form field, and inline `fetch`/`axios`/XHR call, classifies each as `redirect / ssrf / search / id / file / auth / generic` for targeted probes. |
| **JS secrets** | Crawls linked + inline scripts, scans for AWS/GCP/GitHub/Slack/Stripe/Azure/Twilio/Firebase/JWT/private-key tokens with redacted evidence and vendor-file down-weighting. |
| **Template scan** | Optional **nuclei** (medium/high/critical) pass with JSONL parsing; severity counts are back-filled into the attack-surface risk score. |
| **Vuln research** | NVD API keyword search per service/version; optional **searchsploit** hint when available. |
| **Exploitation** | **Off by default** (recon-only). With **`--allow-exploit`**, sequential attempts for HIGH/CRITICAL items (**Hydra** where configured, web flows, etc.); **Playwright** screenshots for successful web-related exploits and optional **priority-path** evidence. |
| **Persistence** | **PostgreSQL** for assets/services when reachable; **Redis** for caching; **in-memory** fallback if DBs are down. |
| **Output** | Timestamped **JSON** and **TXT** under `results//`, optional **`latest.json`**, **CSV** / **SARIF** exports, **`run_id`** / **`correlation_id`** in JSON. |

---

## End-to-end pipeline (8 steps)

The orchestrator runs **strictly in order**: each step completes before the next begins. Logs are prefixed with `[Step N/8]`.

### Step 1 — Initialize storage

- Connects to **PostgreSQL** (`POSTGRES_DSN`) and **Redis** (`REDIS_URL`) if available.
- Initializes schema / stores as implemented in `AssetDatabase`.
- If connections fail, falls back to **in-memory** storage so the run can continue.
- Ensures **`results//`** exists for this run’s reports.

### Step 2 — Asset discovery

- Queries **crt.sh** for certificate transparency names under the root domain.
- If **subfinder** is available, merges passive subdomain enumeration.
- For each subdomain, resolves **A** records and creates **IP** assets.
- Discovers **emails** (MX-derived patterns, WHOIS emails, mail-like strings from homepage HTML).
- Persists assets to the database (or memory).

### Step 3 — Service fingerprinting

- For each **IP** asset, probes a **fixed set of TCP ports** (e.g. 21, 22, 25, 80, 443, database ports, etc.) with short banners / handshakes.
- If **nmap** is available, runs an additional **NSE-backed** scan (`NMAP_SCRIPTS`, with fallback from `vuln,vulners` to `vuln` when scripts are missing).
- Stores **Service** rows (host, port, protocol, name, version, confidence).

**Note:** If there are **no IP assets** (e.g. resolution failed or only non-resolving names), this step may report **0 services fingerprinted** even when tools are installed correctly.

### Step 4 — Directory enumeration (ReconEngine gobuster)

- Resolves a **wordlist** from **SecLists**-style paths and profile (`GOBUSTER_WORDLIST_PROFILE`: small / medium / large).
- If **gobuster** is available, runs `gobuster dir` against the live **http/https** base URL for the target (TLS verification relaxed with `-k` where applicable).
- Collects discovered paths for the report (this is the **engine-level** fuzz pass; integrated recon can run another pass in Step 6).

### Step 5 — Priority path checks (ReconEngine)

- Performs **HEAD**, then **GET** if needed, against a built-in list **`PRIORITY_EXPOSURE_PATHS`** (admin, login, backup paths, common panels, etc.).
- Records interesting status codes (2xx, 3xx, 401, 403) for reporting.

### Step 6 — Integrated domain recon (`recon_domain`)

Runs in a worker thread (`asyncio.to_thread`) and produces a **`DomainReconResult`** (serialized as **`domain_recon`** in JSON):

- **DNS:** `nslookup`-style output, nameservers, reverse DNS per IP.
- **HTTP:** response headers, simple **CDN / Cloudflare** heuristics.
- **Per-IP nmap** (unless disabled): version + NSE scripts as configured.
- **Integrated gobuster** (unless `--quick`, `--no-gobuster`, or `ENABLE_GOBUSTER=0` without `--gobuster`).
- **Optional** homepage / **robots.txt** / **sitemap**-derived directory hints when **web content dirs** are enabled.
- **Priority audit** and **verification** passes on interesting paths (when enabled).

### Step 7 — Vulnerability research

- For services with a **non-empty version string**, queries the **NVD CVE API** (with retries on rate limits).
- Builds **`Vulnerability`** records (CVE id, CVSS-derived severity, description, confidence).
- If **searchsploit** is available, may mark **exploit_available** hints for reporting.

### Step 6b–6h — Attack surface enrichments (runs in order, between Step 6 and Step 7)

Every enrichment writes a partial checkpoint the moment it finishes, so a Ctrl+C / `--max-runtime-minutes` stop still preserves its data:

- **6b — `robots.txt` / sitemap** (`recon_robots.run_robots_scan`): fetches `robots.txt` and recursively parses sitemaps for root + subdomains, scores sensitive-keyword paths (`admin`, `.env`, `.git`, `actuator`, etc.) and records them under `robots_scan` / `all_unique_sensitive_paths`.
- **6c — Subdomain takeover** (`recon_takeover.run_takeover_scan`): combines CNAME-chain analysis (NXDOMAIN dangling + cloud provider namespaces across 40+ services) with HTTP fingerprint strings and writes `subdomain_takeover.findings` (CRITICAL for dangling, HIGH for fingerprint match).
- **6d — Web vulnerability scan (root)** (`recon_webvulns.run_web_vuln_scan`): error + boolean-blind + time-based **SQLi**, context-verified **reflected XSS**, **LFI/path traversal**, **SSRF** cloud metadata, exposed `/.git`, `/.env`, `/actuator`, Swagger, etc., plus HTTP **security headers** audit.
- **6e — JS secrets scan** (`recon_jssecrets.run_js_secrets_scan`): crawls `` + webpack/asset manifests, scans for AWS/GCP/GitHub/Slack/Stripe/Azure/Shopify/Twilio/Firebase/JWT/private keys, DB connection strings, etc., with redacted matches and vendor-file down-weighting.
- **6f — Per-subdomain vulnerability scan**: re-runs `run_web_vuln_scan` on up to 25 live subdomains over HTTPS (falling back to HTTP) and stores per-host counts in `subdomain_vulnerabilities`.
- **6g — Entry point crawler** (`recon_entrypoints.crawl_entry_points`): BFS crawl (depth 2, up to 60 pages) that extracts every URL parameter, form input, and inline `fetch`/`axios`/`XMLHttpRequest` usage; classifies params as `redirect / search / file / id / auth / generic` and flags SSRF candidates. Results land in `entry_points` and seed the next step.
- **6h — Open redirect scan** (`recon_openredirect.run_open_redirect_scan`): injects 13 payload variants (absolute, protocol-relative, triple-slash, `@`-confusion, tab/encoded-slash bypass, CRLF, fully-URL-encoded, etc.), then confirms with a second structurally-different canary before reporting (`findings` / `confirmed_findings`).

### Step 7 — Vulnerability research

- For services with a **non-empty version string**, queries the **NVD CVE API** (with retries on rate limits).
- Builds **`Vulnerability`** records (CVE id, CVSS-derived severity, description, confidence).
- If **searchsploit** is available, may mark **exploit_available** hints for reporting.

### Step 7b — Attack surface mapping (`recon_surface.build_attack_surface_report`)

Aggregates DNS (A/AAAA/MX/NS/TXT/CNAME/SOA, SPF/DMARC), WAF signatures, CORS severity, takeover candidates, open-redirect probes, robots/sitemap sensitive flags, optional **WhatWeb**, TLS hints from Step 6, and the counts from 6d / 7 / 7c into a composite **risk score (0–100, LOW/MEDIUM/HIGH/CRITICAL band)**. The whole structure is written as `attack_surface`.

### Step 7c — Nuclei template scan (`recon_nuclei.run_nuclei_scan`)

If `nuclei` is on `PATH` (or reachable via Go bin / WSL fallback), runs medium/high/critical templates against the primary HTTP service with a 300 s budget, parses the JSONL output, and back-fills critical/high counts into the attack-surface risk score.

### Step 8 — Exploitation pass and reporting

- Selects **HIGH** and **CRITICAL** vulnerabilities only.
- For each, dispatches to **`ExploitationEngine`** based on service type (SSH, MySQL, PostgreSQL, Redis, HTTP/S, etc.).
- On success for **web** services, captures a **Chromium** screenshot via **Playwright**.
- Writes **`report_.json`** / **`.txt`** / **`.html`** under `results//`, including **emails**, **directories**, **priority paths**, **domain_recon**, attack surface, web vulns, nuclei findings, takeover / robots / JS secrets / entry points / open redirects, and **compromised_credentials** summaries when applicable.

---

## Architecture overview

```mermaid
flowchart LR
  subgraph inputs [Inputs]
    ENV[".env"]
    DOM["domains.txt / -t"]
  end
  subgraph core [recon.py]
    ORCH[Orchestrator]
    RE[ReconEngine]
    VR[VulnResearch]
    EE[ExploitationEngine]
    RP[ReportingEngine]
  end
  subgraph data [Storage]
    PG[(PostgreSQL)]
    RD[(Redis)]
    MEM[In-memory fallback]
  end
  subgraph ext [External CLIs]
    NMAP[nmap]
    GB[gobuster]
    SF[subfinder]
    HY[hydra / WSL]
    SS[searchsploit / WSL]
  end
  ENV --> ORCH
  DOM --> ORCH
  ORCH --> RE
  ORCH --> VR
  ORCH --> EE
  ORCH --> RP
  RE --> PG
  RE --> RD
  RE --> MEM
  RE --> NMAP
  RE --> GB
  RE --> SF
  EE --> HY
  VR --> SS
  RP --> OUT["results/..."]
```

---

## Prerequisites

- **Python 3.10+** (3.12+ tested in development).
- **Network** egress for crt.sh, NVD, target hosts, and optional APIs.
- **Legal authorization** for every target in scope.

---

## Installation

### 1. Clone or copy the project

```powershell
cd C:\path\to\recon2exploit
```

### 2. Python dependencies

```powershell
py -3 -m pip install -r requirements.txt
py -3 -m playwright install chromium
```

### 3. Environment file

```powershell
copy .env.example .env
```

Edit `.env` with your PostgreSQL DSN, Redis URL, and preferences (see [Configuration](#configuration-env)).

### 4. PostgreSQL

Create a database (example for Windows default install path):

```powershell
& "C:\Program Files\PostgreSQL\18\bin\psql.exe" -h 127.0.0.1 -U postgres -d postgres -c "CREATE DATABASE asset_db;"
```

Set `POSTGRES_DSN` in `.env` accordingly.

### 5. Redis (optional but recommended)

On Windows, **Memurai** or another Redis-compatible server on `127.0.0.1:6379` works. Point `REDIS_URL` at it.

### 6. Wordlists (SecLists)

Install [SecLists](https://github.com/danielmiessler/SecLists) so default paths resolve, for example:

- `C:\SecLists\...` (matches built-in candidates in `recon.py`)

Without SecLists, gobuster steps may skip or warn until you pass **`--gobuster-wordlist`** with a valid file.

### 7. External security CLIs

| Tool | Role | Typical Windows setup |
|------|------|-------------------------|
| **nmap** | Port scan + NSE | Install via [nmap.org](https://nmap.org/download.html) or Chocolatey; ensure `nmap.exe` is on `PATH` or under `Program Files (x86)\Nmap` (the script also probes that path). |
| **gobuster** | Directory brute-force | `go install github.com/OJ/gobuster/v3@latest` → binary in `%USERPROFILE%\go\bin` (add to user `PATH`). |
| **subfinder** | Subdomains | `go install -v github.com/projectdiscovery/subfinder/v2/cmd/subfinder@latest` → same `go\bin`. |
| **WhatWeb** | *(Optional)* Web tech fingerprinting | Linux package or Ruby gem; if `whatweb` is on `PATH`, attack-surface mapping includes a fingerprint line for the root host. |
| **nuclei** | *(Optional)* Template-based vuln scan (Step 7c) | `go install -v github.com/projectdiscovery/nuclei/v3/cmd/nuclei@latest` → binary in `%USERPROFILE%\go\bin`. Step 7c auto-skips when not installed. |
| **Hydra** | Optional brute-force modules | Often installed inside **WSL** (`sudo apt install hydra` on Ubuntu). `recon.py` can invoke **`wsl -d  -- hydra`** when native `hydra` is missing. |
| **searchsploit** | Exploit-DB hints | Native Linux/Kali, or **WSL** with ExploitDB cloned to `/opt/exploitdb` and executable `searchsploit` there (see [External tools and PATH](#external-tools-and-path)). |

**NSE note:** `vulners` is not always shipped with Nmap. If you see unknown-script errors, set `NMAP_SCRIPTS=vuln` in `.env`, or install the [vulners.nse](https://github.com/vulnersCom/nmap-vulners) script into your Nmap scripts directory.

**Nmap profiles:** Integrated scans and engine NSE use **`NMAP_PROFILE`** (or **`--nmap-profile`**) to choose timing and port scope: `default`, `discovery`, `stealth`, `balanced`, `aggressive`, `quick`, `comprehensive` (see `NMAP_PROFILES` in `recon_surface.py`).

**Attack surface mapping** (after vulnerability research) aggregates DNS (A/AAAA/MX/NS/TXT/CNAME/SOA, SPF/DMARC), WAF hints, CORS, robots/sitemap sensitive-path flags, open-redirect probes, subdomain takeover fingerprints, optional WhatWeb output, and a composite **risk** score (including verified CRITICAL/HIGH CVE counts) into the report field **`attack_surface`**.

---

## Configuration (`.env`)

Variables are loaded from **`.env` in the project directory** (same folder as `recon.py`).

| Variable | Purpose |
|----------|---------|
| `POSTGRES_DSN` | `asyncpg` connection string, e.g. `postgresql://user:pass@127.0.0.1:5432/asset_db`. |
| `REDIS_URL` | Redis URL, e.g. `redis://127.0.0.1:6379`. |
| `DOMAINS_FILE` | Default domains list filename or path (relative paths are resolved from the project directory). |
| `CONFIDENCE_THRESHOLD` | Vulnerabilities below this confidence are omitted from the compact “verified” findings list in the report (default `0.8`). |
| `NMAP_SCRIPTS` | Comma-separated NSE script argument for nmap (default `vuln,vulners`). |
| `NMAP_PROFILE` | Nmap scan profile for integrated + engine scans: `default`, `discovery`, `stealth`, `balanced`, `aggressive`, `quick`, `comprehensive` (overridable with **`--nmap-profile`**). |
| `ENABLE_GOBUSTER` | If `0` / `false` / `no`, integrated gobuster in Step 6 is off unless you pass **`--gobuster`**. |
| `GOBUSTER_WORDLIST_PROFILE` | `small`, `medium`, or `large` — selects SecLists discovery lists when no custom wordlist is set. |
| `GOBUSTER_EXTENSIONS` | Comma list for gobuster `-x` (e.g. `php,txt,html,js,json,xml`). |
| `WSL_DISTRO` | *(Optional)* WSL distribution name for **hydra** / **searchsploit** when using WSL (default `Ubuntu`). |
| `LOG_LEVEL` / `LOG_FORMAT` | Log level and `text` vs **json** lines (see `.env.example`). |
| `CRTSH_DELAY_SEC` | Pause after passive HTTP fetches (polite default). |
| `CERTSPOTTER_API_KEY` / `SECURITYTRAILS_API_KEY` | Optional passive subdomain APIs (respect vendor ToS). |
| `NVD_MAX_CONCURRENT` | Semaphore size for NVD requests (default `1`). |
| `MAX_PARALLEL_HOSTS` | Default parallel IP fingerprint workers (overridable with **`--max-parallel-hosts`**). |

---

## Domains file

- Default file: **`domains.txt`** next to `recon.py`, or whatever **`DOMAINS_FILE`** points to.
- **One hostname or root domain per line.**
- Blank lines and lines starting with **`#`** are ignored.
- When **`-t` / `--target` is omitted**, every non-comment line is scanned **sequentially** (one full pipeline per domain).

---

## Usage and CLI

Use **`--doctor`** or **`--diff-reports`** with no other requirements. For **scans** or **`--plan`**, pass **`--authorized-only`**.

```powershell
cd C:\path\to\recon2exploit

# Diagnostics (no --authorized-only)
py -3 recon.py --doctor

# Compare two report JSON files
py -3 recon.py --diff-reports results\a\report_old.json results\a\report_new.json

# Batch: all domains in DOMAINS_FILE / domains.txt
py -3 recon.py --authorized-only

# Single target + scope allowlist + wall clock cap
py -3 recon.py -t corp.example.com --authorized-only --scope-file scope.txt --max-runtime-minutes 45

# Dry plan (tools/wordlists only, no target I/O)
py -3 recon.py --authorized-only --plan -t example.com

# Recon-only (default): no Step 8 exploits
py -3 recon.py -t example.com --authorized-only

# Enable exploitation attempts after vuln research
py -3 recon.py -t example.com --authorized-only --allow-exploit

# Faster run (skips integrated deep nmap, integrated gobuster, content-derived dirs)
py -3 recon.py --authorized-only --quick

# Disable integrated gobuster only (Step 4 engine gobuster still runs if gobuster is available)
py -3 recon.py --authorized-only --no-gobuster
```

### All CLI flags

| Flag | Effect |
|------|--------|
| `--doctor` | Print diagnostics JSON and exit (no **`--authorized-only`**). |
| `--diff-reports OLD NEW` | Print diff JSON between two reports and exit. |
| `-t`, `--target` | Single root domain or host. If omitted, read **`--domains-file`**. |
| `--domains-file` | Path to domain list (default: env `DOMAINS_FILE` or `domains.txt`). Relative paths are under the project directory. |
| `--authorized-only` | Required for **`--plan`** and normal scans (not for **`--doctor`** / **`--diff-reports`**). |
| `--plan` | Print run plan JSON and exit (no scanning). |
| `--scope-file` | Allowlist path (hostnames + optional CIDR lines). |
| `--max-runtime-minutes` | Per-target timeout (0 = unlimited). |
| `--max-parallel-hosts` | Parallel IP fingerprint cap (0 = default). |
| `--allow-exploit` | Run Step 8 exploit attempts (default: off). |
| `--export-csv` / `--export-sarif` | Write CSV / SARIF after each target’s report. |
| `--no-latest-json` | Skip `latest.json` copy. |
| `--no-priority-screenshots` | Skip Playwright for priority hits. |
| `--quick` | Skips integrated **deep nmap**, integrated **gobuster**, and **content-derived** wordlist behavior. |
| `--no-nmap-deep` | Disables nmap inside integrated `recon_domain` (per resolved IP). |
| `--nmap-profile` | Nmap profile name (default: env `NMAP_PROFILE` or `default`). |
| `--gobuster` | Forces integrated gobuster **on** even if `ENABLE_GOBUSTER=0`. |
| `--no-gobuster` | Forces integrated gobuster **off** (engine Step 4 unchanged). |
| `--gobuster-wordlist` | Absolute or relative path to a custom wordlist for integrated gobuster. |
| `--gobuster-wordlist-profile` | `small` \| `medium` \| `large` when no custom wordlist is set. |
| `--gobuster-extensions` | Overrides comma extensions for gobuster `-x`. |
| `--web-content-dirs` | Forces content-derived prepend on (normally already on for full runs). |
| `--no-web-content-dirs` | Disables homepage / robots / sitemap-derived wordlist hints. |
| `--no-priority-audit` | Skips `PRIORITY_EXPOSURE_PATHS` in integrated recon. |

### “Full” vs `--quick` (integrated recon)

| Feature | Default full run | `--quick` |
|---------|------------------|-----------|
| Integrated per-IP nmap | On | Off |
| Integrated gobuster | On (unless env/flags disable) | Off |
| Content-derived dirs | On | Off |

Steps **1–5** and **7** still run; **6** is lighter when `--quick` is set. **Step 8** runs only when **`--allow-exploit`** is set (otherwise exploitation is always skipped after **7**).

---

## External tools and PATH

- **`resolve_tool_path`** in `recon.py` finds **nmap**, **gobuster**, and **subfinder** via `PATH` **and** common Windows install locations (`%USERPROFILE%\go\bin`, Nmap under Program Files). That avoids “tool missing” when the IDE or service was started with an **outdated** `PATH` after you installed Go tools.
- **`resolve_hydra_argv`** prefers a native `hydra` on `PATH`; otherwise, if **`wsl.exe`** exists and **`/usr/bin/hydra`** is present in the distro **`WSL_DISTRO`**, it runs **`wsl -d  -- hydra ...`** and translates **`-P`** wordlists to **`/mnt/c/...`** paths when needed.
- **`resolve_searchsploit_argv`** prefers native `searchsploit`; on Windows + WSL it can invoke **`/opt/exploitdb/searchsploit`** inside the same distro (install ExploitDB into WSL, e.g. clone GitLab `exploit-database/exploitdb` to `/opt/exploitdb`).

After installing Go-based tools, either **restart** your terminal/IDE or ensure **`%USERPROFILE%\go\bin`** is on your user **`PATH`**.

---

## Reports

Each run writes under:

```text
results//
  report_.json
  report_.txt
  assets_discovered.json          # written right after Step 2 (full asset list, mirrors DB rows)
```

**JSON** (high level):

- `run_id`, `correlation_id`, `target`, `timestamp`, `output_dir`
- `assets` (counts by type), `emails_discovered`, `services_detail`
- `directories_discovered`, `priority_paths`, **`priority_evidence`** (screenshots for priority hits when enabled)
- `vulnerabilities`, `findings` (includes **`remediation`** / **`advisory_url`** per CVE), `exploits`, `compromised_credentials`
- **`domain_recon`**: integrated recon (IPs, nmap, gobuster, priority hits, **`tls_intel`**, **`redirect_chain`**, **`technology_stack`**, errors, etc.)
- **`attack_surface`**: DNS intel, WAF/CORS, takeover candidates, open redirects, robots/sitemap flags, WhatWeb (if installed), **`risk`** score/band (recomputed after nuclei findings are known)
- **`robots_scan`**: per-host `robots.txt` / sitemap parse with sensitive-path scoring (`all_unique_sensitive_paths`).
- **`subdomain_takeover`**: takeover candidates from CNAME chain analysis and HTTP fingerprints, severity per finding.
- **`web_vulnerabilities`**: SQLi / XSS / LFI / SSRF / exposed-file / security-header findings for the primary HTTP host.
- **`subdomain_vulnerabilities`**: the same web scan applied to up to 25 live subdomains (only hosts with at least one finding are kept).
- **`js_secrets`**: redacted high-value secret matches pulled from JS files (+ vendor flag for deprioritised hits).
- **`entry_points`**: crawl-discovered parameters / forms / fetch calls, classified (`redirect`, `ssrf`, `search`, `id`, `file`, `auth`, `generic`).
- **`open_redirect_scan`**: open-redirect findings with `findings` / `confirmed_findings` (confirmed hits are marked CRITICAL).
- **`nuclei_findings`**: parsed nuclei template results (medium/high/critical only) with CVE/CVSS where templates expose them.

**TXT** is a human-readable summary of the same run (truncated lists where applicable). On **cancel** (async cancellation) or **timeout** (`--max-runtime-minutes`), a **`partial_.json`** may be written with the last checkpoint (stage, **`assets`** array after discovery when Step 2 finished, `domain_recon` snapshot, **`attack_surface`** when computed, counts).

**Passive subdomain counts can differ between runs** (crt.sh / subfinder / API timing), so `assets_discovered.json` line counts are not guaranteed identical across reruns.

**`results//latest.json`** mirrors the most recent report JSON for dashboards (disable with **`--no-latest-json`**).

Screenshots for successful web exploits (and optional priority paths) live under **`results//screenshots/`** when Playwright succeeds.

---

## Safety, scope, and ops flags

| Flag / mechanism | Purpose |
|------------------|---------|
| **`--allow-exploit`** | Enables Step 8 exploit attempts. Without it, the pipeline stays **recon-only** after vulnerability research. |
| **`--scope-file`** | Allowlist file: hostnames and optional **CIDR** lines; any target not matching is **dropped** before scanning. |
| **`--plan`** | Prints a JSON plan (resolved tools, wordlist path, passive API config) and **exits** without network scanning. Requires **`--authorized-only`**. |
| **`--max-runtime-minutes`** | Per-target wall-clock cap via `asyncio.wait_for` (entire pipeline). |
| **`--max-parallel-hosts`** | Limits concurrent **IP** fingerprinting workers (default from **`MAX_PARALLEL_HOSTS`** env or `4`). |
| **`--export-csv` / `--export-sarif`** | After each target, writes findings to the given paths (CSV for ticketing, SARIF 2.1.0 for SARIF consumers). |
| **`--no-priority-screenshots`** | Skips Playwright captures for priority-path hits. |
| **`--diff-reports OLD NEW`** | Compares two report JSON files (new subdomains/endpoints, new CVEs) and prints JSON to stdout. |
| **`recon.py --doctor`** | Environment diagnostics (Python version, PATH tools, SecLists probe, `go\bin` probe). No authorization flag. |
| **`LOG_FORMAT=json`** | One JSON log line per record (see `.env.example`). |

Helper modules (imported by `recon.py`):

- **`recon_parsing.py`** — CLI output parsers (nmap, gobuster).
- **`recon_platform.py`** — scope allowlist, HTTP retries, TLS hints, `--doctor`, `--diff-reports`, correlation ID / logging setup.
- **`recon_exports.py`** — CSV, SARIF 2.1.0, and HTML report writers.
- **`recon_surface.py`** — attack-surface aggregation: full DNS, WAF, CORS, takeover heuristics, open-redirect probe, robots/sitemap intel, whatweb, composite risk score.
- **`recon_robots.py`** — robots.txt + sitemap scraper with sensitive-keyword scoring.
- **`recon_takeover.py`** — CNAME dangling + HTTP fingerprint subdomain-takeover detector (40+ services).
- **`recon_webvulns.py`** — async SQLi / XSS / LFI / SSRF / sensitive-file / security-header scanner with multi-stage verification.
- **`recon_entrypoints.py`** — BFS crawler that extracts and classifies every URL parameter / form field / JS fetch call.
- **`recon_openredirect.py`** — 13-payload open-redirect scanner with second-canary confirmation and same-origin FP filtering.
- **`recon_jssecrets.py`** — JS-file crawler + regex secret scanner (AWS/GCP/GitHub/Slack/Stripe/Azure/Twilio/etc.).
- **`recon_nuclei.py`** — nuclei integration (PATH / Go bin / WSL fallback) with JSONL parsing.

---

## Development and CI

- **Tests**: `pip install -r requirements.txt -r requirements-dev.txt` then **`pytest`** (parsing, scope, report shape).
- **Lint / types**: **`ruff check`** on `recon.py`, `recon_*py`, `tests` and **`mypy`** per `pyproject.toml`.
- **GitHub Actions**: `.github/workflows/ci.yml` runs install, ruff, mypy, and pytest on **Python 3.11 and 3.12**.
- **Windows installer script**: `scripts/install-tools.ps1` (Chocolatey **nmap**/**golang**, **`go install`** gobuster/subfinder, PATH hints, WSL reminder).

---

## Troubleshooting

| Symptom | Likely cause | What to do |
|---------|----------------|------------|
| `gobuster not in PATH` / skipped fuzz | `gobuster.exe` not installed or `PATH` stale | Install with `go install`, add `%USERPROFILE%\go\bin` to user `PATH`, restart shell; `recon.py` also probes `go\bin` directly. |
| `services fingerprinted: 0` | No **IP** assets or no open ports on probed set | Check DNS resolution and firewalls; confirm subdomains resolve. |
| NSE / `vulners` errors | Script not installed | Set `NMAP_SCRIPTS=vuln` or install vulners NSE. |
| PostgreSQL / Redis errors at start | Wrong DSN / service down | Fix `.env`; orchestrator falls back to memory but you lose cross-run persistence. |
| Playwright errors | Chromium not installed | Run `python -m playwright install chromium`. |
| Hydra / searchsploit never run on Windows | No native binary and WSL not set up | Install tools in WSL; set `WSL_DISTRO` if your distro is not `Ubuntu`. |
| Long **nmap** / **hydra** runs after **Ctrl+C** | Child process was still reading stdout | Use latest `recon.py`: subprocess reads use cancel-safe cleanup (**terminate** / **kill** on cancellation). You should see `Stopping subprocess pid=…` in the log. |
| Windows **`ConnectionResetError` / WinError 10054** in `_call_connection_lost` during Step 3 | Remote hosts RST idle TCP probes (CDN / WAF); asyncio Proactor logs it | Harmless noise while scanning; `recon.py` filters it to **debug** and uses short connect/read timeouts plus **transport.abort()** on Windows after each probe. |

**Stopping a run:** Press **Ctrl+C** once; the process exits with code **130** after logging `Interrupted by user (Ctrl+C)`. External tools launched via asyncio (**nmap**, **subfinder**, **hydra**, **searchsploit**) are asked to stop when the asyncio task is cancelled.

---

## Legal use

You must have **written permission** and a clear **scope** (hosts, IPs, methods, timing) for every engagement. Unauthorized scanning or exploitation is illegal in most jurisdictions. This software is provided for **defensive and authorized testing** only; you are solely responsible for compliance with applicable laws and contracts.