## https://sploitus.com/exploit?id=KITPLOIT:POSTS-GITHUB-AKANTHED-SECUREAI-SCAN-V060
# SecureAI-Scan
       
**The AI security scanner that proves its findings.**
SecureAI-Scan finds LLM, MCP, Agent Skill, and RAG vulnerabilities in **TypeScript, JavaScript, and Python** β and shows you the evidence: the exact source β flow β sink path for every dataflow finding, resolved through real imports, not keyword matching.
It provides launch-week support for the official OWASP Top 10 for LLM Applications 2026, alongside the Top 10 for Agentic Applications (2026) and the MCP Top 10. Every threat model distinguishes static coverage from runtime concerns.
## Get started in 30 seconds
root@kitploit:~
npx --yes some-email@example.com scan .
No account, cloud upload, Python interpreter, or configuration required. TypeScript, JavaScript, Python, MCP configs, and Agent Skill bundles are detected automatically.
**Measured`0.9.0` release candidate:** 136/136 tests Β· 88.08% statement coverage Β· 12,676 files across 9 public repositories Β· 0 new default-tier fingerprints against the reviewed baseline. Evidence Β· methodology and limits
root@kitploit:~
β HIGH AI001 Prompt injection via user input
PROVEN LLM01:2026 Prompt Injection
source src/chat.ts:8 request data `req.body.input`
flow src/chat.ts:13 passed as `systemPrompt`
sink src/chat.ts:10 openai.chat.completions.create β system role (OpenAI)
fix Keep system prompts static; pass user input as a user-role message.
**Is this for you?** SecureAI-Scan is scoped deliberately to LLM, MCP, and RAG/agent risks β prompt injection, tool poisoning, unsafe output handling, vector-store access control, agent-skill poisoning. It is not a general SAST or secrets scanner, and doesn't try to be one; a known-malicious package with no LLM-shaped payload (e.g. a hardcoded exfiltration address in an email API call) is caught by the offline advisory list (`DEP003`), not a pattern rule. If your codebase talks to an LLM, an MCP server, a vector store, or ships Agent Skills, this is built for you.
## Contents
* Why this scanner is different
* How it compares
* Get started in 30 seconds
* See it work
* Commands
* GitHub Action
* Rules
* Architecture
* MCP server (use it from Claude)
* Claude Skill
* Trust and release assurance
* The precision contract
* Testing & benchmarking
* Roadmap
* Contributing
## Why this scanner is different
* **Evidence tiers, not noise.** Every finding is `proven` (traced dataflow or parsed config fact), `likely` (resolved sink, one heuristic hop), or `heuristic`. **A default scan shows only proven + likely.** Heuristics are opt-in via `--paranoid`.
* **Import-resolved detection.** A call is only an "LLM call" if it resolves to a real SDK import (`openai`, `@anthropic-ai/sdk`, `ai`, `@google/genai`, LangChain, Bedrock, β¦). Your Google Maps client will never be flagged as an LLM again.
* **Precision-gated, and benchmarked against real repos.** The test suite asserts every vulnerable fixture fires _and_ every safe fixture stays clean β a false positive on the safe corpus fails the build. Beyond that, `npm run regression` scans real public repos (OpenAI/Anthropic/Vercel AI SDKs, official MCP servers, LlamaIndex) against a committed, hand-reviewed baseline and **fails on any new`proven`/`likely` finding**. See Testing & benchmarking for the actual before/after numbers, or What we found scanning real repos for the story behind them β a 6/6 catch rate on a labeled malicious-skill corpus, and why we're _not_ calling llama_index "vulnerable" over an honest library-level finding.
* **SARIF for GitHub code scanning.** `--output report.sarif` puts findings inline on pull requests and in the Security tab.
* **AI-BOM.** `secureai-scan bom .` builds a syntax-derived inventory of SDKs, model IDs, vector stores, agent frameworks, and MCP servers, mapped to OWASP LLM Top 10 / EU AI Act documentation needs.
* **MCP config scanning.** Parses `.mcp.json`, `claude_desktop_config.json`, `.cursor/mcp.json`: unpinned `npx -y` servers, inline secrets, plaintext HTTP transports.
* **MCP tool-poisoning detection.** Catches the pattern behind the WhatsApp MCP rug-pull and postmark-mcp backdoor β invisible Unicode, agent-directed injection phrases, and cross-tool shadowing in tool names/descriptions, statically, before you ever run the server.
* **MCP command-injection detection.** Flags MCP stdio transport `command`/`args` built from request data β the pattern behind the 2026 MCP STDIO RCE disclosure.
* **Agent Skill poisoning detection.** The same invisible-Unicode, injection-phrase, and shadowing checks applied to `SKILL.md` files β Agent Skills load into context wholesale, so a poisoned skill is a poisoned tool description by another name.
* **Evasion-resistant skill scanning.** Skill bundles are scanned as _directories_ , not just their `SKILL.md`, and every content check runs against deobfuscated variants of the text. This targets the published techniques β homoglyphs, zero-width splitting, payloads staged in `.git/` or `build/`, exfiltration hidden in a `*.test.ts` file β that bypassed **> 90% of the nine scanners** surveyed in _Cloak and Detonate_ (arXiv:2607.02357). See Evasion resistance.
* **Known-vulnerable and known-malicious package advisories, version-aware.** Checks every dependency and every MCP-launched package against a bundled advisory snapshot β a hand-curated list of documented in-the-wild backdoors, plus HIGH/CRITICAL OSV advisories for an LLM/MCP/RAG package watchlist, regenerated by `scripts/sync-advisories.js`. Runs offline on every scan, no flag required. A CVE only fires when your pinned version is _provably_ inside the affected range; a documented-malicious package fires even on an ambiguous range, because installing a backdoor is unrecoverable.
* **Local-first.** Nothing leaves your machine.
## How it compares
SecureAI-Scan is not a replacement for a general SAST tool or a container/IaC scanner β run it alongside one, not instead of one. It is purpose-built for the LLM/MCP/RAG attack surface and emphasizes dataflow evidence over flat keyword findings.
| SecureAI-Scan| Semgrep (OSS rules)| Trivy| GitHub Advanced Security
---|---|---|---|---
Prompt injection (sourceβsink traced)| β
import-resolved dataflow| β οΈ pattern rules only, community-maintained| β| β οΈ CodeQL can, but no AI-specific ruleset
MCP tool-poisoning / config risk| β
MCP007β010, config scanner| β| β| β
Agent Skill poisoning (`SKILL.md`)| β
evasion-resistant, bundle-aware| β| β| β
RAG / vector-store misconfig| β
VEC001β004| β| β| β
Known-malicious AI package advisories| β
DEP003, offline, version-aware| β| β οΈ general CVE feed, not AI-specific| β οΈ Dependabot, general CVE feed
General SAST (SQLi, XSS, path traversal)| β out of scope by design| β
| β| β
Container / IaC scanning| β| β| β
| β οΈ via CodeQL/Actions
Evidence tiers (proven/likely/heuristic)| β
| β findings are flat| β| β οΈ CodeQL has some, not AI-tuned
SARIF output (GitHub code scanning)| β
| β
| β
| native
Runs offline, no account| β
| β
(OSS rules)| β
| β requires GitHub
If you already run Semgrep or GHAS, keep them β add SecureAI-Scan for the risk surface they don't model at all.
> Prefer to ask questions first? Try the free **SecureAI-Scan AI Security Advisor on ChatGPT**.
> About to run an MCP server you found on GitHub or Twitter? Paste its tool description into **MCP X-Ray** first β checks it for hidden Unicode, injected instructions, and known-malicious packages in your browser, no install.
## See it work
Attack shapes the scanner traces end to end:
MCP tool-poisoning dataflow| RAG context-injection dataflow
---|---
| 
## Commands
The one you need 95% of the time:
root@kitploit:~
secureai-scan scan .
Everything else is there when you need it. `secureai-scan scan . --help` shows all of this in the terminal, grouped the same way:
**Everyday**
Flag| What it does
---|---
_(none)_| `proven` \+ `likely` findings β the default, no flags needed
`--paranoid`| also include `heuristic`-tier findings
`-s, --severity <level>`| only show findings at/above `low`|`medium`|`high`|`critical`
`--output <file>`| write a full report β `.sarif` (GitHub code scanning), `.json`, `.md`, or `.html`
**Scope which rules run**
Flag| What it does
---|---
`-r, --rules <list>`| run only these rule IDs, e.g. `AI001,MCP007`
`--only-ai` / `--only-mcp` / `--only-vec` / `--only-skl`| run only one rule category
`--check-dependencies`| also check `package.json`/`requirements.txt` against the npm/PyPI registry for typos and hallucinated packages (`DEP001`/`DEP002`). Auto-enabled if you select those rules directly via `-r` β you never need to remember to pass both. Not needed for `DEP003` (known-malicious packages), which always runs offline
**CI / workflow**
Flag| What it does
---|---
`--fail-on <severity>`| exit `1` if findings at/above this severity exist
`--baseline <file>`| track only new/changed issues against a saved baseline
`--policy <file>`| load thresholds, skipped paths, and blocked rules from a `.secureai-policy.json` (auto-detected if present β `secureai-scan init` creates one)
**Advanced**
Flag| What it does
---|---
`--min-confidence <0-1>`| finer-grained than `--paranoid`: hide findings below an exact confidence score (`0.9` proven / `0.65` likely / `0.35` heuristic)
`--limit <n>`| max rule groups shown in the terminal (default `10`) β full detail always goes to `--output`
`--debug`| print every file scanned and which rules ran
**Scan before you install β no clone, no config:**
root@kitploit:~
secureai-scan skill anthropics/skills # a GitHub "owner/repo" shorthand
secureai-scan skill https://github.com/β¦ # or a full git URL
secureai-scan skill ./some/local/skill-dir # or a local path
secureai-scan mcp some-mcp-server-package # a bare npm package name
secureai-scan mcp owner/mcp-server-repo # or git, same as `skill`
`skill` and `mcp` fetch the target and scan it, then delete the fetched copy (`--keep` to inspect it instead). Nothing fetched is ever executed: an npm target is downloaded with `npm pack` β the tarball only, no `install`, no lifecycle scripts β and a git target is a plain `git clone --depth 1`. This is the moment that matters most: before a skill lands in `~/.claude/skills/` or a server lands in `.mcp.json`, not after.
**Other commands:**
root@kitploit:~
secureai-scan bom . --output AI_BOM.md # AI Bill of Materials
secureai-scan explain AI001 # why + exploit + fix example, for any rule
secureai-scan threat-model . # THREAT_MODEL.md with the OWASP coverage matrix
secureai-scan init # policy file + CI workflow, one-time setup
Suppress a reviewed finding in code:
root@kitploit:~
// secureai-ignore AI001: reviewed, input sanitized via allowlist
## GitHub Action
root@kitploit:~
name: SecureAI-Scan
on: [pull_request]
permissions:
contents: read
security-events: write
jobs:
scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: akanthed/some-email@example.com
with:
scanner-version: 0.9.0
fail-on: high
Findings appear as inline annotations on the PR and in the repository's Security tab. (`secureai-scan init` generates an equivalent workflow using the CLI directly.)
Scanning clean? Add the badge to your own README:
root@kitploit:~
[](https://github.com/akanthed/SecureAI-Scan)
## Rules
**39 rules** , mapped to the official OWASP Top 10 for LLM Applications (2026) β plus, where applicable, the OWASP Top 10 for Agentic Applications (2026, ASI), the OWASP MCP Top 10 (2025), and an EU AI Act article. See the versioned 2026 coverage and limits; `threat-model` renders the matrix for each scanned project.
Rule| What it proves| OWASP
---|---|---
AI001| User input flows into a system/developer prompt (traced source β sink, including across function/file boundaries)| LLM01
AI002| Prompt content or secrets written to logs (in files that use an LLM SDK)| LLM02
AI003| LLM call in a request handler with no auth check before it| LLM06
AI004| Whole user/session object serialized into a prompt (field-picking is not flagged)| LLM02
AI005| LLM output reaches eval/exec/SQL/HTML sinks| LLM10
AI006| High-impact tools (delete, pay, deploy, β¦) exposed without an approval gate| LLM03
AI007| Retrieved RAG content interpolated into privileged prompts| LLM01
AI008| Secrets embedded in system prompt text| LLM08
AI009| Unbounded user input / missing token limits| LLM06
AI010| Fetched external content flows into prompts| LLM01
AI011| Agent output elevated to system-role in downstream calls| LLM03
AI012| LLM output parsed without schema validation| LLM10
MCP001| MCP tool metadata reaches the system prompt without validation| LLM01
MCP002| MCP server URL constructed from user input| LLM04
MCP003| MCP tool results elevated to system-role| LLM10
MCP004| MCP server launched as an unpinned `npx -y` package| LLM04
MCP005| Secret inlined in a committed MCP config| LLM02
MCP006| MCP server over plaintext HTTP| LLM04
MCP007| Invisible/bidi Unicode hidden in MCP tool names or descriptions| LLM01 Β· MCP03
MCP008| Agent-directed injection phrases in MCP tool descriptions| LLM01 Β· MCP03
MCP009| A tool description that steers calls to a different tool (shadowing)| LLM01 Β· MCP03
MCP010| MCP stdio server command/args constructed from user input (RCE)| LLM04 Β· MCP05
SKL001| Invisible/bidi Unicode anywhere in an Agent Skill bundle| LLM01
SKL002| Agent-directed injection phrasing in a skill's description or body (matched through obfuscation)| LLM01
SKL003| A skill's content steers when/how a different skill is used (shadowing)| LLM01
SKL004| Staged/self-extracting payload: opaque blob + instructions to decode and run it| LLM04 Β· MCP04
SKL005| Credential read + hardcoded external egress in a bundle companion file| LLM02 Β· MCP04
SKL006| Load-time command execution via Claude Code's dynamic-context-injection syntax (`!`cmd``/```!), before any tool-permission gate| LLM04 Β· MCP05
SKL007| Unscoped `Bash` grant in a skill's `allowed-tools` frontmatter| LLM03
SKL008| Skill fetches instructions from an external URL and directs the agent to follow them ("Circus of Skills")| LLM04
SKL009| Skill persists a backdoor by writing into another context file (`MEMORY.md`/`SOUL.md`/`AGENTS.md`/`CLAUDE.md`)| LLM05
SKL010| Unsafe YAML/JSON deserialization tag in a skill's frontmatter or a bundled config file| LLM04
VEC001| Vector search without a tenant/user filter| LLM09
VEC002| Unbounded or user-controlled search limit| LLM06
VEC003| User content ingested into a shared vector store| LLM05
VEC004| Ingestion without tenant/namespace tagging| LLM09
DEP001| Dependency name not found in the registry (opt-in `--check-dependencies`)| LLM04
DEP002| Dependency name one edit away from a popular package (opt-in)| LLM04
DEP003| Dependency with a documented malicious release or critical CVE β checked offline on every scan, version-range aware (postmark-mcp, mcp-remote CVE-2025-6514, β¦)| LLM04 Β· MCP04
`secureai-scan explain <RULE_ID>` gives the exploit walkthrough and a before/after code example for any rule.
## Architecture
Three independent scanning surfaces feed one merged, deduped finding list:
root@kitploit:~
βββββββββββββββββββββββ
*.ts / *.js ββββΆ β ts-morph AST rules βββββ
β (import-resolved β β
β sinks + dataflow) β β
βββββββββββββββββββββββ β
β
βββββββββββββββββββββββ β ββββββββββββββββ βββββββββββββββββββ
*.py ββββΆ β tree-sitter AST + βββββΌββββΆ β scan.ts βββββΆ β evidence filter β
β local taint flow β β β merge/dedupeβ β β confidence β
βββββββββββββββββββββββ β β + suppress β β β severity β
β β (// secure- β β β baseline diff β
.mcp.json, βββββββββββββββββββββββ β β ai-ignore) β β β report β
SKILL.md ββββΆ β Config/bundle scan ββββ ββββββββββββββββ βββββββββββββββββββ
β (off-disk, evasion- β β
β resistant) β βΌ
βββββββββββββββββββββββ terminal Β· sarif Β· json Β· md Β· html
package.json,
requirements.txt ββΆ dependency-guard.ts (advisories.ts, offline, version-aware)
Every AST rule only calls a function an "LLM call" if it resolves through real imports to a known SDK β never by name-matching alone. See `docs/Architecture.md` for the full breakdown of each surface, and `docs/DetectionEngine.md` for how the evidence-tier contract works.
## MCP server (use it from Claude)
The package ships an MCP server exposing `scan_repository`, `explain_rule`, `generate_bom`, and `scan_untrusted_target` (fetch and scan a skill or MCP server before Claude recommends installing it β same fetch-without-executing behavior as the `skill`/`mcp` CLI commands):
root@kitploit:~
{
"mcpServers": {
"secureai-scan": {
"command": "node",
"args": ["/path/to/secureai-scan/mcp-server/index.js"]
}
}
}
## Claude Skill
For Claude Code / Claude.ai users, `skills/secureai-scan/SKILL.md` teaches Claude when to run a scan (reviewing AI/LLM code, or checking an MCP server/Agent Skill before you install it) and how to read the results β no separate process to run, unlike the MCP server above. Copy the `skills/secureai-scan/` directory into your `.claude/skills/` to use it.
## Evasion resistance
In July 2026, two pieces of public research showed that Agent Skill scanners are trivially bypassed.
**Cloak and Detonate** (Ji, Xu, Li, Gao, Wei, Wang & Cheung β HKUST, arXiv:2607.02357) ran an adversarial study against **nine** published skill scanners using 1,613 in-the-wild malicious skills. Two attack families, both of which preserve the payload exactly:
* **Reify** β rewrite the bytes a rule matches, while the agent still reassembles the original at runtime. **> 80% bypass** on most static scanners.
* **Disperse / SFS packing** β move the payload outside the rule's scope: into a directory the scanner skips, behind a renamed extension, or archived and encoded into `.git/`, with a benign cover `SKILL.md` that rebuilds it on first run. **β₯90% bypass on every scanner tested** , β₯96% on most. One hybrid scanner's detection rate fell from ~99% to ~10%.
Separately, **Gecko Security** (VentureBeat) shipped a payload in a `*.test.ts` file. Every public scanner skipped it β but Jest and Vitest auto-discover and execute those files on the next `npm test`.
SecureAI-Scan handles each published technique. Every row below has a corresponding test in `test/deobfuscate.test.js` or a fixture in `test-fixtures/vulnerable/skills/`:
Technique (paper's name)| Example| How it's handled
---|---|---
Zero-width insertion| `Ignβore all preβviousβ¦`| Text is matched against invisible-stripped variants
Confusable glyphs| `Ρurl` (Cyrillic `Ρ`)| Homoglyph folding to ASCII before matching
Case flip| `EvAl`| All content patterns are case-insensitive
Variable splicing| `'cu' + 'rl'`| Adjacent string literals joined before matching
Per-line splitting| `ev`β`al(x)`| Intra-word line breaks joined as an extra variant
Per-file extraction| payload in a sourced helper| Whole bundle is scanned, not just `SKILL.md`
Directory relocation| `scripts/x.sh` β `build/x.sh`| No directory skip-list inside a bundle
Extension renaming| `x.sh` β `x.txt`| Files are scanned by content, not extension
Size padding| pad past the scanner's cap| Oversized files are head-read, never skipped
SFS packing| payload in `.git/skillpack.dat`| Any non-git file under a bundle's `.git/` is `proven` (SKL004)
Test-file staging| payload in `*.test.ts`| Bundle scanning deliberately does **not** demote test paths (SKL005)
### This does not weaken the precision contract
Deobfuscation is usually a precision risk β more matches, more noise. Here the logic is inverted: **a match that appears only after deobfuscation is promoted to`proven`, not demoted.** Ordinary documentation does not contain a zero-width joiner inside "ignore previous instructions", nor a Cyrillic `Ρ` inside `curl`. The concealment is itself affirmative evidence of intent.
The comparison is against the _set_ of raw matches, not merely "did the raw text match at all" β otherwise an attacker could mask the signal by leaving one innocuous phrase in the clear.
The two new bundle rules fire only on conjunctions, never on a keyword:
* **SKL004** needs an opaque blob **and** an unpack directive that _references that blob by name_ β a README mentioning `tar -x` next to an unrelated binary asset is not enough. Real archives (gzip/zip/png/pdf/wasm β checked by magic bytes, not extension) are never "opaque" in the first place, however they're compressed.
* **SKL005** needs a concrete credential _signal_ β a path (`~/.aws/credentials`, not the word "token") or bulk environment-variable enumeration (`os.environ.items()`, not `os.environ["API_KEY"]`) β **and** egress to a hardcoded non-local host, **within 25 lines of each other in the same file** , or a remote fetch that gets executed after being reassigned through one or more renames. A publish helper that reads `~/.npmrc` in one function and calls a registry forty lines later stays clean, and reading one named env var for an API call is never flagged β both shapes are pinned as safe fixtures.
Validated against two real-world corpora, not just fixtures we wrote ourselves: **0 findings across all 18 real skill bundles inanthropics/skills** and all 14 in vercel/ai, and **6/6 correct oncisco-ai-defense/skill-scanner's labeled eval corpus** (20 skills, each with an `_expected.json` verdict) with zero false positives on anything labeled safe. See Testing & benchmarking.
### What this is not
Honest limitation: the paper's conclusion is that runtime detonation beats static analysis, and that is correct. An adaptive adversary who knows these rules can write a transformation they don't cover. What changes here is the _cost_ of evasion β the published, currently-circulating techniques no longer work, and the obfuscation needed to defeat them now itself raises the finding's severity. **Static scanning is a filter, not a security boundary.** Treat an untrusted skill as untrusted code regardless of what any scanner says.
## Trust and release assurance
* CI runs on Linux, Windows, and macOS across supported Node versions.
* CodeQL, production dependency audit, OpenSSF Scorecard, Dependabot, and this scanner's own blocking self-scan provide independent checks.
* Every manual npm publication invokes tests, coverage floors, the reviewed real-repository regression gate, and tarball inspection through `prepublishOnly`.
* GitHub Actions receives no npm password or token and cannot publish the package.
* Release assurance, single-maintainer governance, security reporting, and versioned benchmark evidence are public.
This is a single-maintainer project with no contractual SLA or independent certification. The controls above reduce risk; they do not turn a static scan into proof of security.
## The precision contract
False positives kill scanners. SecureAI-Scan's rule engine follows three hard rules:
1. **Sinks are resolved through imports.** If an identifier resolves to a module that is not an LLM SDK, it is definitively not an LLM call β no matter what it's named.
2. **Evidence is labeled, never blended.** A traced dataflow and a word-proximity match are not the same thing, so they never share a tier.
3. **The safe corpus gates every release.** `test-fixtures/safe/` contains the patterns that used to cause false positives (redacted PII payloads, Google Maps clients, env-var API keys next to LLM clients, ordinary response logging, OAuth metadata fields, streaming-response `chunks`, fiction/narrative prompt text). Any finding there fails the suite.
## Testing & benchmarking
Three layers, because one alone isn't enough to trust a scanner's claims β precision and recall are different failure modes, and both get checked.
**1\. Fixture corpus β precision + recall, runs on every build.**
root@kitploit:~
npm test
`test-fixtures/vulnerable/` and `test-fixtures/safe/` are scanned together: every vulnerable fixture must fire its expected rule at `proven`/`likely` evidence (recall), every safe fixture must produce **zero** `proven`/`likely` findings (precision). Fast and deterministic β but it only proves the scanner behaves on code written specifically to test it.
**2\. Real-world regression benchmark β against public repos we didn't write.**
root@kitploit:~
npm run regression # scan the full curated repo set
npm run regression -- --fresh # re-clone everything first
npm run regression -- openai-node # scan just one repo by name
npm run regression -- --update-baseline # accept the current findings
`scripts/regression-scan.js` clones a curated, diverse set of real public repos (OpenAI/Anthropic/Vercel AI SDKs, the official MCP servers and TypeScript SDK, LlamaIndex, plus anthropics/skills and cisco-ai-defense/skill-scanner for skill-bundle coverage β spanning TS and Python, SDK-consumer example code and SDK-author source) and scans each with the built CLI.
It **exits non-zero on any`proven`/`likely` finding not already in `test/regression-baseline.json`** β a hand-reviewed record of findings already read against their source line. Fingerprints are `repo|rule|file`, not line numbers, so ordinary upstream churn doesn't produce noise. A new fingerprint is a claim the scanner has to justify: if it isn't a genuine issue it's a rule bug, fixed at the root cause and locked in as a new `test-fixtures/safe/` fixture. Baselining a finding you haven't read defeats the entire mechanism.
**Skill-bundle coverage gets its own line** because `cisco-ai-defense/skill-scanner`'s `evals/` corpus is labeled β each of its 20 fixtures ships an `_expected.json` verdict and sits under a directory literally named `malicious/` or `safe/`, so it doubles as a recall check, not just a precision one: **6/6 in-scope malicious fixtures fire, 0 findings on anything labeled safe** , and 0 findings across all 18 real bundles in `anthropics/skills` and all 14 in `vercel/ai`. (The remaining Cisco categories β SQL injection, path traversal, resource exhaustion, generic `eval()` of a function argument, a payload deliberately split across four files β are either out of the documented LLM/MCP/RAG scope or beyond same-file conjunction analysis; see the 0.6.0 changelog entry for the specific reasoning on each.)
Historical before/after from the run that drove the original precision fixes (findings at default evidence level, no `--paranoid`):
Repo| Before| After| What was wrong
---|---|---|---
vercel/ai| 773| 1| `examples/`, top-level `tests/`, and hyphenated `ecosystem-tests/`-style directories weren't recognized as lower-trust paths; `chunks` (a common streaming-response variable) was treated as unambiguous RAG evidence
openai/openai-node| 47| 0| Same path-detection gap, applied to the SDK's own `examples/`/`ecosystem-tests/`
anthropics/anthropic-sdk-typescript| 2| 0| Same path-detection gap on a top-level `tests/` directory
modelcontextprotocol/typescript-sdk| 3| 0| `token_endpoint`/`tokenType`-style OAuth metadata fields flagged as leaked secrets
run-llama/llama_index| 18| 15| A Python check flagged any `description=` field containing "system prompt" as `proven` MCP tool poisoning, regardless of context. The remaining 15 are `VEC001` hits on the library's own generic retriever definitions β scanning a vector-DB SDK's own source, not application code, so a filter can't exist to check; an honest, inherent limit, not a bug
**Current run (2026-08-06)** β versioned evidence is recorded in `docs/benchmarks/v0.9.0.json`:
Repo| Findings| Rules| Status
---|---|---|---
openai-node, anthropic-sdk-typescript, anthropic-sdk-python, modelcontextprotocol/typescript-sdk, modelcontextprotocol/servers| 0| β| clean
anthropics/skills (18 real skill bundles)| 0| β| clean β pure precision check for SKL001β005
vercel/ai (5,691 files)| 0| β| **was 40** (AI001, AI003, AI005, AI010, MCP002) before triage β every one hand-reviewed against source and confirmed a false positive, traced to 3 independent root-cause bugs (see below), fixed, and re-confirmed clean on a full re-scan
run-llama/llama_index| 46| VEC001| inherent limit, not a bug β the library's own generic retriever definitions, where no tenant filter can exist to find
cisco-ai-defense/skill-scanner| 7| SKL001, SKL002, SKL005| **all on fixtures labeled`malicious/`** β 6/6 in-scope, 0 on anything labeled `safe/`
The `vercel/ai` triage found three real, root-caused bugs β none specific to the v0.6.0 skill rules, all in shared logic used across many rules:
1. **`resolveLlmSink` treated any call resolved to an LLM SDK module as a model invocation, regardless of method name** β flagging `isToolUIPart` (a type guard the `ai` package exports right alongside `generateText`) as an LLM call. This alone caused 3 of the 5 finding groups (AI001, AI003, AI010).
2. **`DANGEROUS_CALLEES` in AI005 includes `"query"` for SQL-injection-style sinks, but `"query"` is also a legitimate LLM/agent invocation verb** β `claudeSdk.query({ prompt, options })`, the Claude Agent SDK's own model call, was flagged as "LLM output passed to a dangerous sink" purely because of the shared method name.
3. **`REQUEST_SOURCES` (duplicated identically across MCP002, MCP010, VEC003) matched a bare `"params."`** β any function parameter conventionally named `params`, not necessarily HTTP request data. A URL-scheme validator (`assertOpenLinkParams(params: unknown)`) got flagged as "MCP server URL from user input."
All three fixed at the root cause (not the specific call site) and pinned as permanent fixtures under `test-fixtures/`. Full details in `CHANGELOG.md`.
**3\. Vulnerable-vs-patched validation β proves recall, not just precision.**
The two layers above only check that the scanner stays quiet on safe code. `DEP003`'s advisory checks are validated the other way: pin a package to a documented-vulnerable version and confirm it's flagged, then pin it to the patched version and confirm it isn't.
root@kitploit:~
node --test test/dependency-guard.test.js
covers: `some-email@example.com` (CVE-2025-6514, vulnerable) flagged / `some-email@example.com` (patched) clear; `some-email@example.com` (before the backdoor) clear / `some-email@example.com` (after β no legitimate patch exists for a malicious package) still flagged; `llama-cpp-python==0.2.71` (CVE-2024-34359, from the OSV-generated set) flagged / `==0.2.72` (patched) clear, including under PyPI name normalization (`llama_cpp_python`); and `langchain>=0.1.0`-style unpinned specifiers producing **zero** default-report findings. Building this test caught a real gap: `DEP003` used to match advisories by package name only, never actually comparing the declared version against the advisory's affected range β fixed in `src/scanner/semver.ts`.
Ambiguity is resolved differently per advisory kind, deliberately. A **malicious** package fires even when the declared version can't be resolved β installing a backdoor is unrecoverable, so it fails toward flagging. A **CVE** fires at `proven` only when the declared version is an exact pin provably inside the affected range; unpinned-but-possibly-affected drops to `heuristic` (`--paranoid` only). Applying the malicious-kind rule to a 162-entry CVE snapshot would put a critical finding on every repo that declares `langchain>=0.1.0` β unactionable noise at scale.
## Roadmap
See `ROADMAP.md` for what's shipped and what's planned. Both language engines are AST-based: ts-morph for TypeScript/JavaScript and Tree-sitter for Python. Python imports, calls, assignments, decorators, scopes, keyword arguments, dictionary fields, and strings are syntax nodes; target code is never imported or executed, and no Python interpreter is required. The remaining Python gap is bounded cross-function/cross-file taint depth, not parsing. Scan performance and known limits are documented in `docs/Performance.md`.
## Contributing
Contributions are welcome β see `CONTRIBUTING.md` for the workflow, and `docs/WritingRules.md` / `docs/RuleDevelopment.md` for how to add a detection rule that meets the precision bar above. Every new rule needs a fixture in both `test-fixtures/vulnerable/` and `test-fixtures/safe/`, an entry in `src/scanner/catalog.ts`, and a case in `test/corpus.test.js` β `npm test` enforces all three.
## License
MIT Β© Akshay Kanthed
Read more