Hunter Irrigation Product Scraper

# 🌿 Hunter Irrigation Product Scraper

📋 Overview

A comprehensive, production-ready web scraper built with Selenium and Python that extracts the complete Hunter Irrigation product catalog including detailed specifications, images, documents, support links, and related products. Designed for Magento 2 import with professional data visualization capabilities.

✨ Features

🎯 Complete Data Extraction (48+ Fields)

• Product Information: Names, SKUs, descriptions, categories, brands, models
• Technical Specifications: Radius, flow, pressure, dimensions, trajectory
• Images: Thumbnail, gallery images, application photos, studio photos
• Documents: Manuals, spec sheets, marketing materials, CAD files with download URLs
• Support Resources: Installation guides, FAQs, performance cards, adjustment guides
• Related Products: Compatible products and accessories
• Shipping Information: Quantities, pallet configurations, master carton details

🚀 Advanced Features

• Dynamic Category Detection - Automatically extracts all categories from website navigation
• Smart Retry Logic - Retries failed categories up to 3 times to prevent false negatives
• Image Extraction - Extracts all images from Slick carousel galleries
• Tab Navigation - Clicks through product tabs to extract Features, Documents, Support, Photos, Related products
• Fast & Optimized - Chrome profile caching, eager page loading strategy
• Professional Data Visualizer - Beautiful Flask web app with modern UI to preview scraped data
• Magento 2 Ready - CSV export format fully compatible with Magento 2 import

🚀 Quick Start

Prerequisites

Python 3.8 or higher
Google Chrome browser

Installation

1. Clone the repository

git clone https://github.com/RasikhAli/Hunter-Irrigation-Product-Scraper.git
cd Hunter-Irrigation-Product-Scraper

1. Install dependencies

pip install -r requirements.txt

Usage

1. Test Scraper (Recommended First)

python hunter_scraper_test.py

• Scrapes 3 products from 2 categories
• Takes approximately 2-3 minutes
• Outputs to hunter_products_test.csv
• Perfect for testing and validation

2. View Data in Professional UI

python data-visualizer/app.py

• Open browser to http://localhost:5000
• Beautiful, modern interface with filtering and search
• View all product details, images, and specifications
• Responsive design for mobile and desktop

3. Full Scraper (Production)

python hunter_scraper.py

• Scrapes ALL categories and products
• Takes several hours depending on catalog size
• Outputs to hunter_products_full.csv
• Includes retry logic for reliability

🎨 Data Visualizer

Professional Flask web application with modern, gradient-based UI featuring:

• 📊 Dashboard - Statistics cards showing total products, categories, images, specs
• 🔍 Advanced Filtering - Filter by category, search by name/SKU
• 🖼️ Image Galleries - View all product images including application and studio photos
• 📱 Responsive Design - Works perfectly on desktop, tablet, and mobile
• ⚡ Fast Performance - Optimized rendering with smooth animations
• 🎯 Professional UI - Modern gradients, glassmorphism effects, Font Awesome icons

Screenshots

Catalog View: Grid layout with product cards, images, and quick specs Product Detail: Complete product information with image gallery and all extracted fields

📊 Output

CSV File Structure

• Encoding: UTF-8 with BOM
• Separator: Comma (,)
• Fields: 48 columns including:
  • Basic: sku, name, category, brand, description
  • Images: image_thumbnail, image_gallery, application_photos, studio_photos
  • Specs: radius, flow, operating_pressure, height, diameter
  • Documents: documents (title|URL format)
  • Support: support_links (title|URL format)
  • Related: related_products (name|URL format)
  • Shipping: qty_per_bag, ocean_pallet, land_pallet, master_carton

Data Format Examples

Multiple items separated by: ||
URLs and titles separated by: |
Example: "Manual|https://example.com/manual.pdf||Spec Sheet|https://example.com/spec.pdf"

🔧 Technical Details

Architecture

• Selenium WebDriver - Browser automation for dynamic content
• Custom Chrome Profile - Faster loading with cached resources
• Eager Page Load Strategy - Don't wait for all resources
• Tab Navigation - Clicks through product tabs for complete data
• Retry Mechanism - Up to 3 retries with 5-second delays
• Error Handling - Graceful fallbacks for missing data

Performance

• Test Scraper: ~26 seconds per product (includes tab navigation)
• Full Scraper: Varies by catalog size (typically 4-8 hours for full catalog)
• Memory Usage: ~200-300 MB
• Chrome Profile: Reduces load time by 30-40%

📁 Project Structure

Hunter-Irrigation-Product-Scraper/
├── hunter_scraper.py              # Full production scraper
├── hunter_scraper_test.py         # Test scraper (2 categories, 3 products each)
├── setup_chromedriver.py          # Chrome driver setup utility
├── requirements.txt               # Python dependencies
├── data-visualizer/               # Flask web application
│   ├── app.py                     # Flask application
│   ├── templates/                 # HTML templates
│   │   ├── index.html            # Catalog view
│   │   └── product_detail.html   # Product detail view
│   ├── requirements.txt          # Visualizer dependencies
│   └── run.bat                   # Windows launcher
├── chrome_profile/                # Chrome user data (auto-created)
├── drivers/                       # ChromeDriver (auto-downloaded)
├── hunter_products_test.csv       # Test output
└── hunter_products_full.csv       # Full scraper output

🛠️ Configuration

Modify Test Scraper

Edit hunter_scraper_test.py:

test_categories_limit = 2  # Number of categories to scrape
max_products_per_category = 3  # Products per category

Change CSV Path in Visualizer

Edit data-visualizer/app.py:

CSV_FILE = Path(__file__).parent.parent / 'hunter_products_full.csv'

🐛 Troubleshooting

ChromeDriver Issues: The scraper auto-downloads ChromeDriver. If issues occur, delete drivers/ folder and re-run.

Timeout Errors: Increase wait times in scraper:

time.sleep(3)  # Increase from 1.5 to 3 seconds

Missing Data: Check retry logic is enabled and working. Some products may genuinely not have certain fields.

CSV Encoding Issues: File uses UTF-8 with BOM. Open in Excel or use proper CSV reader.

📝 License

MIT License - Free for personal and commercial use.

👨‍💻 Author

Rasikh Ali