Harfetcher

A professional-grade automation tool to automatically capture and export HAR (HTTP Archive) files from Google Chrome using Selenium and the Chrome DevTools Protocol (CDP). HarFetcher doesn't just capture logs—it intelligently extracts the data into structured formats for easy analysis.

✨ Key Features

• 🚀 Zero-Config Capture: Automatically generates a new HAR file on every page reload or navigation.
• 🛠️ CDP Integration: Leverages Chrome DevTools Protocol to capture full request/response bodies, including JSON APIs and HTML.
• 🧠 Smart Extraction: Automatically parses raw HAR files and organizes content into:
  • API: REST/JSON payloads and endpoint data.
  • Assets: JavaScript, CSS, and media files.
  • Pages: Main HTML documents.
• 🍪 Cache Preservation: Operates without clearing the browser cache, allowing for realistic, stateful browsing sessions.
• 📂 Automated Workspace: Self-initializing folder structure to keep your data organized from the first run.

🏗️ Architecture & How It Works

HarFetcher acts as a background observer for your Chrome session. Here's how the magic happens:

1. CDP Enablement: It enables the Network and Page domains in Chrome's DevTools Protocol.
2. Performance Logging: Selenium captures performance entries which are then reconstructed into a valid HAR 1.2 format by HARExporter.
3. Body Harvesting: For matching MIME types (JSON, Text, JS), it calls Network.getResponseBody to secure the actual payload.
4. Modular Storage: FolderBuilder ensures every snapshot is safely filed away with timestamped identifiers.

📁 Project Structure

HarFetcher/
├── output/                   # Main data repository (Auto-generated)
│   ├── raw_har/              # Original .har files (Full capture)
│   ├── extracted/            # Processed and categorized data
│   │   ├── api/              # API responses and JSON payloads
│   │   ├── assets/           # Scripts, Stylesheets, and Media
│   │   └── pages/            # HTML source of navigated pages
│   └── logs/                 # Detailed application logs (app.log)
├── har_exporter.py           # Core logic for CDP capture and HAR reconstruction
├── folder_builder.py         # Handles directory initialization and pathing
├── main.py                   # Orchestration, Selenium setup, and event loop
├── requirements.txt          # Project dependencies
└── README.md                 # You are here!

🛠️ Installation & Setup

1. Prerequisites

• Python 3.8+
• Google Chrome installed on your system.

2. Setup

Clone the repository and install the required dependencies:

# Create a virtual environment
python -m venv venv

# Activate the venv
# On Windows:
.\venv\Scripts\activate
# On macOS/Linux:
source venv/bin/activate

# Install dependencies
pip install -r requirements.txt

🚀 Usage

Running HarFetcher is straightforward:

1. Start the script:

   python main.py
   

2. Browse: The script will open a Chrome window. Any page you visit, reload, or navigate to will be automatically captured.
3. Review: Check the output/ folder in real-time. You'll see .har files appearing in raw_har/ and extracted content appearing in extracted/.
4. Exit: Close the browser window or press Ctrl+C in the terminal to safely shut down.

⚙️ Configuration

You can easily customize the behavior by editing main.py:

• Start URL: Change driver.get("...") near line 55 to your desired landing page.
• Polling Rate: Adjust time.sleep(3) at the bottom of the loop to change how frequently it checks for new network activity.
• Stabilization Delay: Adjust time.sleep(2) before generate_har to allow complex pages more time to finish loading before extraction.

🛡️ Troubleshooting

• Chromedriver Issues: We use setup-chromedriver to handle pathing, but ensure your Chrome version is up to date.
• Missing Bodies: Some responses (like large binary files or certain secure streams) might not be captureable via CDP getResponseBody.
• Windows Path Lengths: If you encounter Errno 22, HarFetcher automatically hashes very long URLs to prevent Windows path length violations.

📜 License

MIT © 2024 HarFetcher Team