Compass Lead Intelligence Outreach Platform

A high-performance async sitemap-driven web crawler and administrative dashboard designed to collect real estate agent profiles, active property listings, and closed deals (past sales & rentals) from Compass.com.

✨ Features

• 🌐 Sitemap-Driven Traversal: Crawls the official Compass sitemap starting from root states (/sitemap/agents/) down to counties/cities (/sitemap/agents/{state}/{county}/ & /agents/locations/{city}-{state}/{id}/).
• ⚡ Fully Async Architecture: Powered by aiohttp and asyncio with thread-pool offloading for HTML parsing via BeautifulSoup4.
• 🏠 Comprehensive Data Extraction:
  • Agent Profile: Name, Title, Email, Phone, Office Name/Address, State, City, License Number, Biography, Profile Image, Areas Served, Credentials, Languages.
  • Active Listings: Listing URLs, Price, Beds, Baths, SqFt, Property Type, Date Posted, Description.
  • Past Sales & Rentals: Closed deal records (both sales and rentals) extracted directly from agent profile JSON blocks and Compass closed deals APIs.
• 🛡️ Smart Deduplication & URL Normalization: Profile URLs are normalized to prevent duplicate agent records across multiple city coverage areas. When an agent is listed in multiple cities, their profile is updated/merged without duplication.
• 🎨 Glassmorphic Dark UI Dashboard:
  • Live metric counters for Total Agents, Active Listings, Past Sales, and Past Rentals.
  • Real-time Server-Sent Events (SSE) stdout log stream terminal.
  • State-level filter controls and city/state dropdown filters.
  • Clean data table with 15-item pagination, search, and detailed profile modal view.
  • Instant task cancellation stop button.
• 📊 In-Memory Export: One-click download of filtered or full database datasets as formatted .json or .csv (Excel compatible).
• 🗑️ Administrative Data Management: Supports targeted deletion by state, city, or full database purge.

🛠️ Project Structure

CompassScraper/
├── static/
│   ├── app.js             # Frontend controller (SSE, table rendering, modals, filtering)
│   └── style.css          # Dark glassmorphic styling system
├── templates/
│   └── index.html         # Main administrative dashboard UI
├── app.py                 # Flask server, SSE broadcast, API endpoints, background scanner thread
├── db.py                  # SQLite database layer (4 tables: scrape_locations, agents, listings, past_sales)
├── scraper.py             # Sitemap-driven async crawler engine
├── outputcompasssitemap/  # Sitemap HAR and extracted endpoint reference metadata
├── compass_agents.db      # SQLite database file (created automatically)
├── requirements.txt       # Python package dependencies
├── .gitignore             # Full git exclusion specification
└── README.md              # Project documentation

🔒 Database Schema

The SQLite database (compass_agents.db) operates in WAL mode with foreign keys enabled:

1. scrape_locations: Tracks location traversal status (pending, done, error), state, city, and agents found count.
2. agents: Stores agent profile details. Unique constraint on normalized profile_url.
3. listings: Stores active property listings associated with an agent (agent_id FK).
4. past_sales: Stores closed deals (sale & rental transactions) associated with an agent (agent_id FK). Unique constraint on (agent_id, listing_id).

🚀 Quick Start

1. Prerequisites & Virtual Environment

Python 3.10+ is recommended.

# Clone or navigate to the repository
cd CompassScraper

# Activate your virtual environment (Windows example)
.\venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

2. Run the Dashboard

python app.py

Open your web browser and visit:

http://127.0.0.1:5000

3. Start Scraping

1. Enter one or more state names in the Target States field (e.g. Arizona, Texas), or leave blank to scrape all available states across the Compass sitemap.
2. Click Start Scrape.
3. Watch live stdout logs, real-time counters, and auto-updating agent records in the table!