## https://sploitus.com/exploit?id=1867A945-DC35-5454-9940-BEA26E75E2BB
# Talos-Apache-Log-Oversight-Scanner
## Overview
The `Talos-Apache-Log-Oversight-Scanner` is a powerful and efficient Python 3 utility designed for internal Cisco Talos teams to analyze Apache web server access logs (or other HTTP access logs in a similar format) for potential security threats and operational insights. Leveraging the high performance of `ripgrep` for initial data extraction, it then intelligently processes and presents key findings, including various attack patterns, IP activity, and HTTP request statistics.
This tool aims to streamline the process of identifying suspicious activity, understanding traffic patterns, and quickly pinpointing relevant log entries within large datasets, providing crucial **oversight** into web server security.
## Features
* **High Performance:** Utilizes `ripgrep` (`rg`) for ultra-fast, multi-threaded log file scanning, significantly reducing processing time on large volumes of data.
* **Comprehensive Threat Detection:** Identifies a wide range of security-related patterns, including:
* SQL Injection attempts
* Web scanner and reconnaissance tool activity (e.g., Nmap, Burp Suite)
* Requests targeting sensitive paths (e.g., admin panels, config files, database interfaces)
* General exploiting attempts (e.g., null bytes, command execution indicators)
* Web shell presence indicators
* Cross-Site Scripting (XSS) attempts
* Directory/Path Traversal attempts
* Local/Remote File Inclusion (LFI/RFI) attempts
* Command Injection attempts
* Cloud Metadata/Server-Side Request Forgery (SSRF) attempts
* JNDI Injection (e.g., Log4Shell patterns)
* Internal SSRF attempts (targeting private IP ranges)
* Authentication Bypass/Brute Force indicators
* Malicious User-Agent/Bot activity
* **IP Address Analysis:** Lists the top 20 most frequent IP addresses observed.
* **HTTP Status Code Analysis:** Provides counts of the most common HTTP status codes (e.g., 200 OK, 404 Not Found, 500 Internal Server Error).
* **Requested Path Analysis:** Shows the most frequently requested paths, categorized by HTTP status code.
* **Interactive Filtering:** Allows users to filter security matches by specific HTTP status codes (e.g., only show 404-related attacks).
* **Rarest IP Analysis:** Identifies and displays IP addresses involved in the fewest security matches, providing specific examples of the matched log lines and patterns for context. This helps uncover targeted or stealthy attacks.
* **Customizable Scanning:** Include or exclude specific security pattern categories to focus your analysis.
* **IP Exclusion:** Filter out known benign IP addresses from all analysis.
* **Color-Coded Output:** Uses ANSI escape codes for enhanced readability, highlighting matched patterns in red and HTTP status codes in context-specific colors (e.g., 2xx/5xx in white, 3xx in cyan, 4xx in yellow).
* **Progress Indicator:** Displays a progress bar during scanning for large datasets.
## Prerequisites
Before running the script, ensure you have the following installed:
* **Python 3.x**: The script is written in Python 3.
* **ripgrep (`rg`)**: This is a crucial dependency for the script's performance.
* **macOS (Homebrew)**: `brew install ripgrep`
* **Debian/Ubuntu**: `sudo apt-get install ripgrep`
* **Arch Linux**: `sudo pacman -S ripgrep`
* **Fedora**: `sudo dnf install ripgrep`
* For other operating systems, refer to the [ripgrep installation guide](https://github.com/BurntSushi/ripgrep#installation).
## Installation
1. **Clone the repository:**
```bash
git clone https://github.cisco.com/talos/Talos-Apache-Log-Oversight-Scanner.git # Updated repository name
cd Talos-Apache-Log-Oversight-Scanner
```
2. **Make the script executable:**
```bash
chmod +x Tlog_scanner.py
```
## Usage
Run the script from your terminal. Use the `-h` or `--help` flag to see all available options and their descriptions.
```bash
./Tlog_scanner.py --help
```
## Scan a directory:
```bash
./log_scanner.py -d /var/log/apache2/
```
## Scan a single file:
```bash
./log_scanner.py -f /var/log/nginx/access.log
```
## List available security patterns:
```bash
./log_scanner.py --list-patterns
```
## Scan a directory, showing top paths and status codes:
```bash
./log_scanner.py -d /var/log/nginx/ --top-paths --top-status
```
## Limit output for top paths/status codes:
```bash
./log_scanner.py -d /var/log/apache2/ --top-status --status-limit 5
```
## Include only specific security patterns:
```bash
./log_scanner.py -d /var/log/ --include-patterns "SQL injection attempts" "XSS attempts"
# Or with comma-separated names in a single string:
./log_scanner.py -d /var/log/ --include-patterns "Sensitive paths,Web shell matches"
```
## Exclude specific IP addresses from analysis:
```bash
./log_scanner.py -d /var/log/ --exclude-ips 192.168.1.1 10.0.0.5
```
## Scan, then interactively filter security matches by HTTP codes (e.g., 404, 500):
```bash
./log_scanner.py -d /var/log/apache2/
# (After initial results, you will be prompted: "Show security matches filtered by HTTP codes (e.g., 200,404) or 'n' to skip:")
# Enter: 404,500
```