## https://sploitus.com/exploit?id=B33A8A99-8B45-5EA7-9323-46A4B09014DA
# Nuclei POC Duplicate Detection Tool
This tool is written in Go language and is used to scan YAML POC files of nuclei. It detects duplicate POCs and automatically moves them to the duplicates folder.
## Features
- π **Smart Duplicate Detection**: Detects duplicate POCs based on HTTP request path information.
- π **Recursive Scanning**: Supports scanning specified directories and their subdirectories.
- π **Automatic Movement**: Automatically moves duplicate POC files to the `duplicates` folder.
- π **Detailed Report**: Displays scan results and information about duplicate files.
- π‘οΈ **Secure Handling**: Avoid overwriting existing files and automatically adds serial numbers.
## Installation Requirements
```bash
go mod tidy
```
## How to Use
```bash
go run main.go -f
```
### Example
```bash
# Scan the nuclei directory
go run main.go -f ./nuclei
# Scan other directories
go run main.go -f /path/to/your/pocs
```
## How It Works
1. **Path Extraction**: Extract HTTP request paths from the YAML filesβ `http.raw`, `http.path`, and `requests.raw` fields.
2. **Path Cleaning**: Remove template variables (e.g., `{{Hostname}}`) and query parameters.
3. **Duplicate Detection**: Detect duplicate POCs based on the cleaned path information.
4. **File Movement**: Move duplicate POC files to the `duplicates` subfolder.
## Supported YAML Format
The program supports standard nuclei template formats, including:
- `http.raw` β Original HTTP requests
- `http.path` β Array of HTTP paths
- `requests.raw` β Original request data
## Output Example
```
Starting scan of the ./nuclei directory.
Found 495 POC files.
66 duplicate POCs found:
Moving duplicate files: nuclei\gitlab\CVE-2021-4191.yaml
Moving duplicate files: nuclei\ruijie\ruijie-nbr1300g-exposure.yaml
... Duplicate files moved to: nuclei\duplicates
```
## Notes
- The program will automatically create the `duplicates` folder.
- If the target file already exists, serial numbers will be added automatically to avoid overwriting.
- The program will skip the `duplicates` folder itself to avoid duplicate processing.
- Duplicate detection is based on HTTP paths; POCs with the same path are considered duplicates.
## Technical Implementation
- Uses `gopkg.in/yaml.v3` to parse YAML files.
- Regular expressions are used to extract HTTP request paths and methods.
- Intelligent path cleaning and deduplication algorithms are implemented.
- Secure file movement operations are performed.