## https://sploitus.com/exploit?id=AB5BCED1-F447-5D52-A6B8-DC1F17AF85D6
# LogSentinel β Intelligent Web Log Security Analysis Platform
A web application based on FastAPI. It allows users to upload or paste web server logs (Nginx/Apache), and automatically identifies common attacks such as brute force attacks, SQL injections, directory traversal, and XSS attacks. Visual security reports are generated accordingly.
## I. Design Concept
### 1.1 Requirements Analysis
Web servers generate a large amount of access logs every day, which may contain various types of attacks. Traditional manual review methods are inefficient and prone to missed threats. This project aims to provide an automated log security analysis tool to help operations and security teams quickly identify potential threats. Key design goals:
- Zero-dependency deployment: No database required, no persistent storage; all data is processed in memory and discarded once used.
- Ready-to-use: Supports both file upload and text paste input methods, compatible with both Common and Combined log formats.
- Visualization prioritized: Provides visual representations of attack trends, making security analysis easier.
- Security consistency: The platform itself also needs protection (e.g., against path traversal, rate limits, etc.).
### 1.2 System Architecture
```
βββββββββββββββββββββββββββββββββββββββββββββββββββββββ
β User Browser β
β ββββββββββββ ββββββββββββ βββββββββββββββββββββ β
β β Home Upload β β Analysis Report β β Export Report (PDF) β β
β ββββββ¬ββββββ ββββββ²ββββββ ββββββββββ²βββββββββββ β
βββββββββΌββββββββββββββΌββββββββββββββββββΌβββββββββββββββ
β POST β HTML β POST
β /analyze β Response β /export
βββββββββΌββββββββββββββ΄ββββββββββββββββββ΄βββββββββββββββ
β FastAPI (main.py) β
β ββββββββββββββ ββββββββββββ ββββββββββββββββββββββ β
β β Rate Limiting β β Exception Handling β β Jinja2 Template Rendering β
β β (slowapi) β β (HTML-friendly)β β (base/report/...) β β
β ββββββββββββββ ββββββββββββ ββββββββββββββββββββββ β
β β β
β ββββββββββΌβββββββββ β
β β LogAnalyzer β β
β β (analyzer.py) β β
β ββββββββββββββββββββ€ β
β β 1. Log Line Parsing β β
β β 2. Attack Pattern Matching ββββ patterns.py β
β β 3. IP Behavior Profiling β (Regular Expression Library) β
β β 4. Timeline/Thermogram β β
β β 5. Security Recommendations β β
β β 6. IP Masking Output β β
β ββββββββββββββββββββ β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
```
### 1.3 Processing Flow
1. Users upload `.log/.txt` files or paste log texts through the home page.
2. The backend extracts log content, performing security checks on file names, size restrictions, and encoding detection.
3. `LogAnalyzer` parses logs line by line (automatically recognizing Common/Combined formats).
4. Multi-dimensional attack detections are performed on each log (regular expression matching + aggregate analysis).
5. IP behavior profiles, attack timelines, and heatmap data are generated.
6. IP addresses in the output are masked (with end-coding).
7. Visual reports are rendered, allowing users to view or export PDF versions online.
---
## II. Technical Highlights
### 2.1 Multi-layer Attack Detection Engine
Adopting a dual-layer detection strategy of "regular expression matching + behavior aggregation":
| Detection Type | Detection Method | Number of Rules | Description |
|-----------------|-------------------|--------------|-------------|
| SQL Injection | Regular Expression Matching | 20 rules | Includes UNION SELECT, OR 1=1, time-based injection, and URL encoding decoding |
| XSS Attacks | Regular Expression Matching | 18 rules | Includes HTML entity encoding bypass, event handlers, and HTML entity encoding bypass |
| Directory Traversal | Regular Expression Matching | 14 items | Overrides `../`, `/etc/passwd`, Unicode encoding bypass |
| Scanner Recognition | UA regular expression + frequency aggregation | 14 UA items + frequency threshold | Identifies tools like sqlmap/nmap/nikto, and detects frequent path scans |
| Brute-force Attacks | Time window aggregation | Threshold >10 attempts/minute | Statistics based on IP + minute window for 401/403 failure counts |
| Detection Duplication Mechanism: Uses `detected_lines` collection to prevent duplicate marking of the same log line by multiple rules. ### 2.2 IP Behavior Profiling | For each IP, create multi-dimensional profiles: - Total requests, error rate (4xx/5xx ratio), - Number of attacks and attack type clustering, - First/first occurrence time, - Distribution of HTTP methods, status codes |
| High-risk IPs are ranked by number of attacks, with the top 20 displayed. ### 2.3 IP Masking | Before analyzing the results, all IP addresses are automatically masked at the end (e.g., `192.168.1.xxx`), covering three dimensions: attack list, high-risk IP ranking, and IP profile, to protect privacy. ### 2.4 Visual Reports | Based on Chart.js 4.x (locally hosted, no external CDN dependencies): - Pie charts of attack types distribution, - Bar charts of attack timelines, - Stacked heatmaps of 24 hours Γ 5 attack types, - Ranking of high-risk IPs (details available), - Detailed tables of attack events (support for filtering by type) |
| 2.5 Report Export (PDF) | This project uses βfront-end printingβ to export PDFs: - Click βExport Reportβ on the report page β Send current analysis results to the `/export` interface, - `/export` returns an optimized HTML page for printing (`report_export.html`), opened in a new tab, - The page includes complete `@media print` styles: forced print background color, pagination control, conversion of Canvas charts to static images, - Users can export by clicking βPrint β Save as PDFβ in their browser. This is the latter option among βweasyprint or front-end printingβ options, with the advantage of having zero server-side dependencies and cross-platform compatibility. ### 2.6 Security Protection Design | Protective Measures | Implementation Methods | Prevention of Path traversal | os.path.basename() + file name character validation |
| File Type Restrictions | Only `.log` / `.txt` allowed | File Size Limit | β€ 5MB | β€ 10,000 lines |
| Rate Limiting | slowapi 3 times/minute/IP | No Persistent Storage | All data processed only in memory | Friendly Error Pages | HTML error pages or JSON based on Accept headers |
| Security Response Headers | CSP, X-Frame-Options, X-Content-Type-Options, etc. | Security Response Header Configuration: - X-Content-Type-Options: nosniff β prevents MIME type sniffing, - X-Frame-Options: DENY β prevents clickjacking, - X-XSS-Protection: 1; mode=block β XSS filtering, - Referrer-Policy: strict-origin-when-cross-origin β Referral policy, - Content-Security-Policy β Content security policy (limits scripts, styles, fonts) |
| 2.7 Full Localization of Front-End Resources | All front-end dependencies (Bootstrap 5.3.3 CSS/JS, Bootstrap Icons, Chart.js 4.4.1) for the main application pages (homepage, report pages) are downloaded to the `static/` directory locally, with no external CDN dependencies, ensuring proper loading in any network environment. > Note: The report export page (`report_export.html`) uses Google Fonts as an external font to ensure proper display of Chinese in PDFs. See details in βSection 7: Report Exportβ. --- |
| 3. Demonstration Screenshots | 3.1 Home Page β Log Uploading and Pasting | Supports dragging and uploading files or pasting log texts directly, with real-time verification of file format and size on the front end.  |
| 3.2 Analysis Report β Overview and Charts | Displays key metrics such as total number of log lines, successful parsing rates, number of attack events, and high-risk IP count, along with pie charts of attack type distribution and bar charts of attack timelines.  |
| 3.3 Analysis Report β Attack Heatmap | A stacked bar chart showing the distribution of attacks over 24 hours for 5 types of attacks, providing a visual representation of attack patterns over time.  |
| 3.4 Analysis Report β High-Risk IP Ranking and Attack Details | High-risk IP rankings allow viewing detailed profiles (request method distribution, status code distribution, first/first occurrence time). Attack event lists can be filtered by type.  |
| 3.5 Analysis Report β Security Recommendations | Automatically generates targeted protection recommendations based on detection results, sorted by severity.  |
| 3.6 Exporting Reports | Click βExport Reportβ to open a report page optimized for printing in a new tab. Export can be done by clicking βPrint β Save as PDFβ in the browser. ! |
[Export Report Screenshot](docs/screenshots/06-export.png)
> Note: If the screenshot is not visible, please run the project and refer to the βDemo Instructionsβ for actual results. ---
## IV. Technology Stack
| Level | Technology | Description |
|------|------|------|
| Backend Framework | Python 3.12 + FastAPI | Asynchronous, high-performance web framework |
| Template Engine | Jinja2 | Server-side rendering of HTML |
| Frontend UI | Bootstrap 5.3.3 | Responsive layout, compatible with PC/tablets |
| Icons | Bootstrap Icons 1.11.3 | Vector icon library |
| Visualization | Chart.js 4.4.1 | Pie charts, bar charts, stacked heat maps |
| Security | slowapi | IP-based rate limiting |
| Deployment | Docker + docker-compose | One-click containerized deployment |
---
## V. Project Structure
```
LogSentinel/
βββ main.py # FastAPI main program (routing, exception handling, log extraction)
βββ analyzer.py # Core analysis engine (parsing, detection, profiling, masking)
βββ utils/
β βββ __init__.py
β βββ patterns.py # Attack pattern ruleset + detection threshold configuration
βββ templates/
β βββ base.html # Base template (navigation bar, footer)
β βββ index.html # Home page (upload/paste)
β βββ report.html # Report analysis page (charts, interactive elements)
β βββ report_export.html # Export report template (print optimization)
β βββ error.html # Friendly error page
βββ static/
β βββ css/
β β βββ bootstrap.min.css
β β βββ bootstrapIcons.min.css
β β βββ style.css # Custom styles
β β βββ fonts/ # Icon font files
β βββ js/
β βββ bootstrap.bundle.min.js
β βββ chart.umd.js # Chart.js 4.4.1 (hosted locally)
β βββ chart.js # Custom chart rendering logic
βββ sample_logs/
β βββ sample_attack.log # Example attack logs
βββ requirements.txt # Python dependencies
βββ Dockerfile # Docker build file
tests/ # Automated testing (54 cases)
βββ conftest.py # Shared fixtures
βββ test_analyzer.py # Analysis engine unit tests
βββ test_api.py # API routing integration tests
test_samples/ # Manual test samples
βββ normal_access.log # Normal access (0 attacks)
βββ mixed_attack.txt # Mixed attack logs (5 types of attacks)
```
---
## VI. Quick Start
### Docker Method (Recommended)
```bash
docker-compose up --build -d
```
Access at http://localhost:8081
### Local Development Method
```bash
cd LogSentinel
pip install -r requirements.txt
python main.py
```
Access at http://localhost:8000
### Configuration Instructions
Detection thresholds can be customized via environment variables or configuration files:
**Via Environment Variables:**
```bash
export LOGSENTINEL_BRUTE_FORCE_THRESHOLD=20 # Brute-force attack threshold (default 10)
export LOGSENTINEL_SCANNER_REQUEST_THRESHOLD=100 # Scanner request threshold (default 50)
export LOGSENTINEL_MAX_LOG_LINES=20000 # Maximum number of log lines (default 10000)
export LOGSENTINEL_MAX_FILE_SIZE_MB=10 # Maximum file size in MB (default 5)
```
**Via Configuration Files:**
```bash
cp LogSentinel/config.example.json LogSentinel/config.json
# Edit config.json to modify settings
```
Configuration priority: Environment variables > Configuration files > Default values
---
## VII. Report Export
### Export Options
The report page offers two export options:
| Button | Description |
|------|------|
| Export as HTML | Open the print-optimized HTML page and export it as a PDF using the browserβs βPrint β Save as PDFβ option |
### Exporting to PDF | The backend generates PDF files for download (requires WeasyPrint installation).
#### Frontend Printing Export (Default)
The default printing scheme is used without additional dependencies:
1. Click the βExport HTMLβ button.
2. Open the printing optimization page in a new tab.
3. Use browser shortcuts `Cmd+P` (Mac) or `Ctrl+P` (Windows) to select βSave as PDFβ.
#### Instructions for Chinese Display in PDFs
For Chinese to be displayed correctly in exported PDFs, Web fonts are required. Since browsers do not embed system fonts when printing to PDF (e.g., Apple Fonts, Microsoft YaHei), Google Fontsβ Noto Sans SC font is used to ensure correct rendering of Chinese. **Note:**
- The export page (`report_export.html`) requires a network connection to load Google Fonts.
- It may take a few seconds to download the font files during the first load.
- If the network is unavailable, the system will revert to using system fonts, which may cause abnormal display of Chinese in PDFs.
- The main application pages (homepage, report page) remain zero-dependency-free; only the export page uses external fonts.
This is a trade-off between βzero external dependenciesβ and βcorrectly displaying Chinese in PDFsβ. For complete offline use, it is recommended to:
1. Use the backend PDF generation solution (WeasyPrint).
2. Pre-load the exported page with cached fonts when connected to a network.
#### Backend PDF Generation (Optional)
If you want the PDF file generated directly by the backend, You need to install WeasyPrint and its system dependencies:
**macOS:**
```bash
brew install pango cairo libffi gdk-pixbuf
pip install weasyprint
```
**Ubuntu/Debian:**
```bash
apt-get install libpango-1.0-0 libpangocairo-1.0-0 libgdk-pixbuf2.0-0 libffi-dev libcairo2
pip install weasyprint
```
**Docker Deployment:** Modify the `Dockerfile` to include system dependencies, then rebuild the container. After successful installation, click the βExport PDFβ button to download the PDF file directly. If WeasyPrint is not installed, the system will automatically revert to the frontend printing scheme, and the user will be prompted. You can check the availability of WeasyPrint via the `/api/config` interface:
```bash
curl http://localhost:8081/api/config
# {"config": {...}, "weasyprint_available": true/false}
```
---
## VIII. Testing
### Automated Testing
The `tests/` directory at the project root contains 55 test cases, covering both the analysis engine and API routing.
```bash
pip install httpx pytest pytest-asyncio
python -m pytest tests/ -v
```
**Test Structure:**
| File | Test Content | Number of Cases |
|------|--------------|-------------|
| `tests/test_analyzer.py` | Log parsing, SQL injection/XSS/directory traversal/scanner/vulnerability detection, IP profiling, IP masking, heatmap, security recommendations, full analysis process | 41 |
| `tests/test_api.py` | Homepage rendering, text analysis, file upload, export interfaces, HTML/JSON error handling, security checks, configuration API | 14 |
### Manual Test Cases
The `test_samples/` directory contains two test files that can be uploaded to the platform for analysis:
| File | Description | Expected Result |
|------|--------------|------------------|
| `test_samples/normal_access.log` | 20 normal access logs | 0 attack events; report should state βNo threats detectedβ |
| `test_samples/mixed_attack.txt` | 35 mixed logs (normal + 5 types of attacks) | SQL injection, XSS, directory traversal, vulnerability detection detected |
Additionally, `LogSentinel/sample_logs/sample_attack.log` contains a more comprehensive 65-line example of an attack log. ---
## IX. Demonstration Instructions
1. Visit the homepage and upload the sample log file `LogSentinel/sample_logs/sample_attack.log` (or paste the log text in the βPaste Logβ tab).
2. Click βStart Analysisβ, and the system automatically identifies and detects attacks such as SQL injection, brute force attacks, XSS, directory traversal, and scanners.
3. View visual charts: pie charts showing attack type distribution, bar charts for time-line analysis, and heatmaps for 24-hour attacks.
4. View a ranking of high-risk IPs; click to expand detailed IP profiles.
5. View a list of attack event details, filter by attack type.
6. View targeted security protection recommendations.
7. Click the βExport Reportβ button; open the printing optimization page in a new tab, and use βPrint β Save as PDFβ to export the report.
---
## X. API Interfaces
| Method | Path | Description | Parameters |
|------|------|------|------|
| GET | / | Render the homepage | - |
| POST | /analyze | Receive logs and analyze them, return the report page | `file` (uploaded file) or `log_text` (pasted text) |
| POST | /export | Export the report | `log_text` + `format` (html/pdf) |
| GET | /api/config | Get current configuration | - |