📑 Table of Contents

• ✨ Key Features
• 🏗️ Architecture
• 🧰 Tech Stack
• 📸 Dashboard Preview
• ⚡ Quick Start
• 🔧 Installation
• 🚀 Running the Application
• 🧠 How It Works
• ⚙️ Configuration
• 📁 Project Structure
• 🗄️ Database Schema
• 🔌 API Reference
• 📤 Data Export
• ⏰ Scheduled Scraping
• 🛡️ Troubleshooting
• 🤝 Contributing
• 📜 License
• 🙏 Acknowledgments

✨ Key Features

| Feature | Description | |---|---| | 🔌 CDP-Powered Scraping | Connects to your own Chrome browser over the Chrome DevTools Protocol on port 9222 — no headless fingerprinting, dramatically lower detection rates. | | 🧩 Adaptive Partitioning | Intelligently splits massive search results (year × make × model × door × cylinder) and runs partitions in parallel for sub-hour runs on 100k+ listings. | | ⚡ Two Fetch Engines | API Mode uses browser-context fetch() for detail pages (≈ 10× faster). Traditional Mode navigates separate tabs for maximum compatibility. | | 🔁 Smart Deduplication | Merges with existing records, tracks price history across runs, and never re-fetches fully processed listings. | | 📊 Live Dashboard | Glassmorphic, real-time UI powered by Socket.IO — listings stream in as they're scraped, with run statistics, filters, and export. | | ⏰ Built-in Scheduler | Auto-scrape every N hours with live countdown. Set-and-forget data collection. | | 📤 Excel + JSON Export | One-click export of filtered or complete datasets with clean column formatting and control-character sanitization. | | 🛡️ Captcha-Aware | Detects Cloudflare challenges, waits for human resolution, and resumes automatically once cleared. | | 🧱 Resource Blocking | Drops images, fonts, media, and tracker scripts for ≥ 3× speedup on listings pages. | | 🧪 Quality Filters | Sanitizes cylinders/doors to reject garbage (e.g. UNIX timestamps) inherited from legacy data. | | 💾 Crash-Safe Storage | SQLite with WAL journal mode, batched commits every 50 inserts or 1 second. | | 🔄 Legacy Migration | Auto-migrates the original cars_cumulative.json into SQLite on first run. |

🏗️ Architecture

                ┌────────────────────────────────────────────────────────┐
                │                   Dashboard (SPA)                     │
                │   templates/dashboard.html  •  Socket.IO Client        │
                └─────────────────────┬──────────────────────────────────┘
                                      │  WebSocket (status, new_car)
                                      ▼
┌──────────────────────────────────────────────────────────────────────────────┐
│                       Flask + Flask-SocketIO Server                          │
│                                app.py                                         │
│  ┌────────────┐  ┌──────────────┐  ┌──────────────┐  ┌──────────────────┐   │
│  │ REST API   │  │  Scheduler   │  │  Chrome Mgr  │  │  Message Broker  │   │
│  │ /api/...   │  │  (daemon)    │  │  (CDP 9222)  │  │  queue.Queue     │   │
│  └─────┬──────┘  └──────┬───────┘  └──────┬───────┘  └────────┬─────────┘   │
│        └────────────────┴─────────────────┴───────────────────┘             │
│                                       │                                      │
│                                       ▼                                      │
│                       ┌──────────────────────────────┐                      │
│                       │       CarsScraper Class      │                      │
│                       │          scraper.py           │                      │
│                       │  • Playwright async engine    │                      │
│                       │  • Partition planner         │                      │
│                       │  • Detail-page fetcher        │                      │
│                       │  • Cloudflare detector        │                      │
│                       └──────────┬───────────────────┘                      │
│                                  │  CDP connect_over_cdp                    │
│                                  ▼                                          │
│              ┌──────────────────────────────────────┐                       │
│              │  Google Chrome (User's Browser)      │                       │
│              │  --remote-debugging-port=9222        │                       │
│              │  cars.com/shopping/results/...       │                       │
│              └──────────────────────────────────────┘                       │
│                                  │                                          │
│                                  ▼                                          │
│                       ┌──────────────────────────────┐                      │
│                       │       SQLite  cars.db        │                      │
│                       │       db.py  (WAL mode)      │                      │
│                       └──────────────────────────────┘                      │
└──────────────────────────────────────────────────────────────────────────────┘

🧰 Tech Stack

Backend

• 🐍 Python 3.10+ — Core language
• 🌶️ Flask 3.0 — HTTP server
• ⚡ Flask-SocketIO 5.3 — Real-time bidirectional events
• 🎭 Playwright (async) — Browser automation over CDP
• 🗄️ SQLite (WAL) — Embedded persistence
• 🐼 pandas + openpyxl — Excel export

Frontend

• 🅱️ Bootstrap 5.3 — Layout
• 🎨 Bootstrap Icons — Iconography
• 🔤 Inter & Outfit — Premium typography
• ✨ Vanilla CSS — Custom glassmorphism design system

Browser

• 🟢 Google Chrome / Chromium — Headful CDP target

📸 Dashboard Preview

The dashboard is a single-page, real-time interface with a premium dark theme:

• 🎯 Interactive 5-Step Stepper — Visualizes the workflow (Launch Chrome → Open Link → Connect → Scrape → Export)
• 📈 Live Stat Cards — Total cars, current run, scheduled next run, average scrape time
• 🔍 Instant Search — Filter by make, year, title, or VIN with debounced live results
• 📑 Sortable Pagination — Server-side pagination with sortable columns
• 🪟 Detail Modal — Full vehicle history, dealer info, price history timeline
• ⏱️ Timing History — Last 50 scrape runs with per-listing averages
• 📊 Active Partition Tracker — See which (year, make, model) is being scraped right now

⚡ Quick Start

Prerequisites: Python 3.10+, Google Chrome installed.

# 1. Clone the repo
git clone https://github.com/RasikhAli/Car.com-Scraper.git
cd Car.com-Scraper

# 2. Create a virtual environment
python -m venv .venv
# Windows
.venv\Scripts\activate
# macOS / Linux
source .venv/bin/activate

# 3. Install dependencies
pip install -r requirements.txt
playwright install chromium

# 4. Run the server
python app.py

Now open http://localhost:5001, click Ready Chrome Debugger, paste the URL into the auto-launched Chrome, and press Start Scraping. 🎉

🔧 Installation

1. System Requirements

| Component | Minimum | Recommended | |---|---|---| | Python | 3.10 | 3.11+ | | RAM | 2 GB | 4 GB+ | | Chrome | 110+ | Latest stable | | Disk | 500 MB | 2 GB (for SQLite growth) | | OS | Windows 10 / macOS 12 / Ubuntu 20.04 | Latest LTS |

2. Install Python Packages

pip install -r requirements.txt

The requirements.txt includes:

Flask==3.0.0
Flask-SocketIO==5.3.4
python-socketio==5.10.0
python-engineio>=4.8.0
selenium==4.18.0
requests==2.31.0
pandas==2.2.0
openpyxl==3.1.2
undetected-chromedriver==3.4.6
opencv-python==4.9.0.80
pyautogui==0.9.54
numpy==1.26.4
playwright

3. Install Playwright Browsers

playwright install chromium

💡 On Linux/VPS you may also need: playwright install-deps

4. (Optional) Manually Launch Chrome in Debug Mode

If you prefer to use your existing Chrome profile:

Windows

"C:\Program Files\Google\Chrome\Application\chrome.exe" --remote-debugging-port=9222

macOS

/Applications/Google\ Chrome.app/Contents/MacOS/Google\ Chrome --remote-debugging-port=9222

Linux

google-chrome --remote-debugging-port=9222 --no-sandbox

🚀 Running the Application

python app.py

The server binds to 0.0.0.0:5001. Visit:

• Dashboard: http://localhost:5001
• Status JSON: http://localhost:5001/api/status
• Cars JSON: http://localhost:5001/api/cars

First-Run Behavior

If cars.db is empty, the app automatically migrates any legacy cars_cumulative.json file in the project root.

🧠 How It Works

The scraper follows an elegant 5-step pipeline that mirrors the on-screen stepper:

1. 🔌 Launch Chrome — App spawns a fresh, isolated Chrome instance with --remote-debugging-port=9222. Uses an isolated user-data-dir (%TEMP%/cars_scraper_profile) so it never collides with your normal profile.

2. 🌐 Navigate to Filter URL — A clean https://www.cars.com/shopping/results/?... URL is built from your filter selections (year, makes, zip, distance, sort).

3. 🔗 Connect via CDP — playwright.chromium.connect_over_cdp(...) attaches to the existing tab — no headless detection, no new browser instance.

4. 🧮 Adaptive Partitioning — The engine checks the page count for each (year, make) combination. If ≤ 100 pages, it's scraped as one partition. Otherwise, it splits by model, then by door count × cylinder count, running partitions in parallel (default batch = 4).

5. 📡 Stream to Dashboard — Each scraped listing is pushed through queue.Queue → SQLite (batched commit) → Socket.IO → live UI insert. No page reloads. No polling.

Scraping Modes

| Mode | Behavior | Best For | |---|---|---| | 🚀 Load All First (default) | Phase 1: Collect every unique listing URL. Phase 2: Parallel detail fetches. Zero duplicates, faster overall. | Large datasets, fresh runs | | 🔄 Scrape As You Go | Streams listings page-by-page in real time. | Watching the run live, partial-stop resumes |

Fetch Methods

| Method | Description | Speed | |---|---|---| | ⚡ API Mode | Uses browser-context fetch() to hit cars.com's internal detail endpoint directly. | ~10× faster | | 🌐 Traditional | Opens each listing in a separate Playwright tab. | Compatible with all layouts |

⚙️ Configuration

All configuration is driven by Python constants in app.py and runtime filters in the dashboard.

Default Filters (app.py)

filters = {
    'year_min': '2021',
    'makes': ['BMW', 'Land Rover', 'Mercedes-Benz'],
    'zip': '60606',
    'maximum_distance': '9999',
    'fetch_method': 'api',
    'scraping_mode': 'scrape_as_you_go',   # or 'load_all_first'
    'batch_size': 4,                       # concurrent partitions
}

Scheduling (app.py)

scheduling = {
    'enabled': False,
    'interval_seconds': 43200   # 12 hours
}

You can change these at runtime via the dashboard's Auto-Scrape panel.

CDP Port

Default: 9222. Change CDP_PORT in app.py if you need a different port.

📁 Project Structure

Car.com-Scrapper/
│
├── app.py                 # Flask + SocketIO server, REST API, scheduler
├── scraper.py             # CarsScraper class (Playwright async, partitioning)
├── db.py                  # SQLite schema, migrations, CRUD, exports
├── requirements.txt       # Python dependencies
├── .gitignore             # Git ignore rules
├── README.md              # ← you are here
│
├── templates/
│   └── dashboard.html     # Single-page glassmorphic dashboard (Bootstrap 5)
│
├── static/                # (reserved for future static assets)
│
├── cars.db                # SQLite database (auto-generated, gitignored)
├── cars_cumulative.json   # Legacy data file (auto-migrated on first run)
│
├── cars_export.json       # Generated on demand via /api/export?format=json
└── cars_export.xlsx       # Generated on demand via /api/export?format=excel

🗄️ Database Schema

cars — Main vehicle table (50+ columns)

| Column | Type | Description | |---|---|---| | listing_id | TEXT PK | Primary key from cars.com (or composite fallback) | | title | TEXT | e.g. "2022 BMW X5 xDrive40i" | | price | TEXT | Current dealer price | | link | TEXT | Direct URL to the listing | | page | INTEGER | Result page it was discovered on | | vin | TEXT | Vehicle Identification Number | | mileage | TEXT | Odometer reading | | msrp | TEXT | Manufacturer's Suggested Retail Price | | exterior_color / interior_color | TEXT | — | | features | TEXT (JSON) | Array of feature strings | | description | TEXT | Dealer description | | year, make, model, trim | TEXT | — | | body_style, fuel_type, stock_type, condition | TEXT | — | | cpo | INTEGER (0/1) | Certified Pre-Owned flag | | seller_zip, seller_customer_id | TEXT | — | | trid, delivery_type, financing_type, isa_context | TEXT | Internal tracking fields | | primary_thumbnail | TEXT | Image URL | | search_score, ship_price | TEXT | — | | metadata | TEXT (JSON) | Raw metadata blob | | monthly_payment, monthly_label | TEXT | Financing estimate | | seller_notes | TEXT | — | | dealer_info | TEXT (JSON) | {name, rating, address, phone, ...} | | drivetrain, transmission, engine, cylinders, doors | TEXT | — | | vehicle_history | TEXT (JSON) | Accident count, owner count, etc. | | deal_rating | TEXT | Great / Good / Fair / etc. | | location, dealership_link | TEXT | — | | price_history | TEXT (JSON) | [{price, date}, ...] | | scraped_at | INTEGER | Unix timestamp of last update |

scrape_runs — Run history (audit + analytics)

| Column | Type | Description | |---|---|---| | id | INTEGER PK AUTOINCREMENT | — | | pages_scan_start / pages_scan_end | INTEGER | Unix timestamps | | scrape_start / scrape_end | INTEGER | Unix timestamps | | avg_scrape_time_per_listing | REAL | Seconds per listing | | listings_scraped | INTEGER | Total listings processed | | mode | TEXT | load_all_first or scrape_as_you_go | | created_at | INTEGER | Unix timestamp |

Indexes & Optimizations

• PRAGMA journal_mode = WAL — Concurrent readers + single writer
• 30-second connection timeout — Survives long scrapes
• Batched commits — Every 50 inserts or 1 second
• Automatic rollback — On insert failure, the connection is recycled
• Allowed sort columns (whitelist) — scraped_at, price, mileage, year, make, model, monthly_payment (SQL-injection-safe)

Legacy Migration

On first run, if cars.db is empty, the app auto-migrates the legacy cars_cumulative.json file:

def init_db(legacy_json_path=None):
    # ... creates schema ...
    if count == 0 and legacy_json_path and os.path.exists(legacy_json_path):
        with open(legacy_json_path, 'r') as f:
            data = json.load(f)
            for car in data.get('results', []):
                insert_or_update_car(conn, car)

🔌 API Reference

REST Endpoints

| Method | Endpoint | Description | |---|---|---| | GET | / | Dashboard SPA | | GET | /api/status | Current scraper state, filters, schedule | | GET | /api/cars?page=&limit=&search=&make=&year=&sort_by=&sort_order= | Paginated car list | | POST | /api/launch_chrome | Launch debug-Chrome and return the target URL | | POST | /api/delete_car | Delete one car (by listing_id) | | POST | /api/delete_by_make | Delete all cars of a make | | POST | /api/delete_by_year | Delete all cars of a year | | POST | /api/delete_all | Wipe the entire database | | GET | /api/export?format=json\|excel&filter_type=&make=&year= | Download filtered export | | GET | /api/timing_history | Current + historical run timings | | GET / POST | /api/timer_config | Get / set auto-schedule interval | | GET | /favicon.ico | Empty 204 |

WebSocket Events (Socket.IO)

| Event | Direction | Payload | |---|---|---| | connect | client → server | (none) — server emits initial status_update | | start_scraping | client → server | {scraping_mode, fetch_method, batch_size} | | stop_scraping | client → server | (none) | | update_filters | client → server | {year_min, makes, ...} | | toggle_scheduling | client → server | {enabled, interval_hours} | | status_update | server → client | Full status payload (auto-emitted on changes) | | new_car | server → client | Single car record | | scrape_reset | server → client | Emitted at start of each run | | scrape_complete | server → client | {status, count} | | scrape_stopped | server → client | {status} | | scrape_error | server → client | {error} | | filters_updated | server → client | New filter dict |

Example: Fetch Latest 50 Cars

curl 'http://localhost:5001/api/cars?page=1&limit=50&sort_by=scraped_at&sort_order=desc'

Example: Export All BMWs to Excel

curl -o bmw_export.xlsx 'http://localhost:5001/api/export?format=excel&filter_type=make&make=BMW'

Example: Start a Scrape via WebSocket

import socketio
sio = socketio.Client()
sio.connect('http://localhost:5001')
sio.emit('start_scraping', {'scraping_mode': 'load_all_first', 'batch_size': 8})

📤 Data Export

The app generates on-demand exports in two formats:

JSON Export

GET /api/export?format=json[&filter_type=all|make|year|make_year&make=BMW&year=2023]

Returns a JSON file with all columns, properly deserialized JSON fields (features, dealer_info, price_history, etc.).

Excel Export (.xlsx)

GET /api/export?format=excel[&filter_type=...&make=...&year=...]

Generates a clean .xlsx with:

• ✅ All 50+ columns flattened
• ✅ JSON fields serialized to strings (Excel-readable)
• ✅ Control characters stripped (no file corruption)
• ✅ Cell text length-capped at 32,700 chars per cell
• ✅ Trailing ... [TRUNCATED] marker on overflow

Filter Types

| filter_type | Required Params | Effect | |---|---|---| | all | (none) | Export every car | | make | make=BMW | Export only BMW | | year | year=2023 | Export only 2023 | | make_year | make=BMW&year=2023 | Export only 2023 BMWs |

⏰ Scheduled Scraping

Enable Auto-Scrape from the dashboard's "Auto-Scrape" panel.

Defaults

• Interval: 12 hours (43,200 seconds)
• Configurable range: 1 second to 168 hours (1 week)
• Persistence: Interval value lives in app.py; toggle state is in-memory

Programmatic Control

# In app.py
scheduling = {
    'enabled': False,
    'interval_seconds': 43200   # 12 hours
}

# Via WebSocket
sio.emit('toggle_scheduling', {'enabled': True, 'interval_hours': 6})

# Via REST
curl -X POST http://localhost:5001/api/timer_config \
     -H "Content-Type: application/json" \
     -d '{"interval_hours": 6}'

Behavior

• ✅ Live countdown displayed in dashboard header
• ✅ New run automatically starts on schedule
• ✅ Scheduler uses a daemon thread + threading.Event (no busy-loop)
• ✅ Skips run if a scrape is already in progress
• ✅ Browser tab is kept open between scheduled runs

🛡️ Troubleshooting

SELECT vin, COUNT(*) c FROM cars GROUP BY vin HAVING c > 1;

pip install playwright
playwright install chromium

🤝 Contributing

Contributions are warmly welcome! Whether it's a bug fix, new feature, or documentation improvement.

Development Setup

# Fork and clone
git clone https://github.com/RasikhAli/Car.com-Scraper.git
cd Car.com-Scraper

# Create branch
git checkout -b feature/amazing-thing

# Install dev dependencies
pip install -r requirements.txt
playwright install chromium

# Make your changes...

# Run the app
python app.py

Code Style

• Python: PEP 8, type hints encouraged
• Frontend: Bootstrap utility classes + custom CSS variables (defined in :root)
• Commits: Conventional Commits (feat:, fix:, docs:, refactor:)

Areas for Contribution

• 🧪 Unit tests for db.py and partition logic
• 🌐 Multi-site support (autotrader.com, carmax.com, etc.)
• 🤖 ML price-prediction based on scraped history
• 📱 Mobile dashboard with reduced layout
• 🐳 Docker image for one-command deploys
• 📊 Grafana / Prometheus exporter for scrape metrics
• 🔐 CAPTCHA solver integration (2Captcha, AntiCaptcha)

Submitting a PR

1. Update the README with any new features
2. Add/extend docstrings
3. Test manually with a small filter (1 make, 1 year)
4. Open the PR with a clear description & screenshots

📜 License

Distributed under the MIT License. See LICENSE for the full text.

MIT License

Copyright (c) 2026 Rasikh Ali

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

⚠️ Disclaimer: This project is for educational and personal use only. Scraping cars.com may violate their Terms of Service. The authors are not responsible for any misuse. Always respect the target site's robots.txt and rate limits.

🙏 Acknowledgments

This project stands on the shoulders of giants:

• 🎭 Playwright — Powerful, modern browser automation
• 🌶️ Flask & Flask-SocketIO — Elegant real-time web framework
• 🗄️ SQLite — The world's most deployed database
• 🐼 pandas & openpyxl — Seamless Excel export
• 🅱️ Bootstrap & Bootstrap Icons — Beautiful, responsive UI primitives
• 🔤 Inter & Outfit — Premium open-source fonts
• 🌐 Chrome DevTools Protocol — The bridge that makes this whole project possible