Rental Property Scraping
Relevant source files
The following files were used as context for generating this wiki page:
The Rental Property Scraping system is a specialized module designed to collect, normalize, and aggregate housing cost data from various international real estate portals. Unlike the grocery comparison system, this module treats listings as ephemeral data points used to compute pooled medians rather than reusable product entities. It supports multi-country targets, including Ukraine and Australia, and handles complex currency conversions and regional normalization.
Sources: src/database/migrations/015_create_rental_tables.sql:1-4, src/scrapers/rent/RentScraperService.ts:33-40
System Architecture
The architecture consists of three primary phases: extraction, normalization, and aggregation. The system utilizes Playwright for browser automation on specific portals and custom parsers for JSON-based data exchange formats.
Data Flow Overview
The following diagram illustrates how rental data moves from external websites into the system's aggregated statistics.
The flow starts with external extraction, followed by normalization to local currencies, and ends with periodic aggregation into statistical buckets.
Sources: src/scrapers/rent/RentScraperService.ts:47-87, scripts/aggregate-rent.ts:16-56
Scraper Implementation
The system uses a modular approach where specific scrapers are assigned to countries and cities. Each scraper is responsible for navigating to the target portal and returning a standardized ListingRaw object.
Supported Sources and Targets
The system tracks specific cities through the RENT_TARGETS configuration, mapping them to country codes and respective scrapers.
| Country | Sources | Target Cities |
|---|---|---|
| Ukraine (UA) | olx, domria, flatfy | Kyiv, Lviv, Odesa |
| Australia (AU) | realestateau, domainau | Sydney, Melbourne, Brisbane |
Sources: src/scrapers/rent/RentScraperService.ts:22-31, src/scrapers/rent/rentTargets.ts
Browser Automation and Extraction
For complex sites like RealEstate.com.au, the system uses Playwright to bypass basic detection and extract data from internal state objects like window.ArgonautExchange.
The scraper service includes a "headed fallback" mechanism where it retries the scrape in a visible browser if headless mode fails to return results.
Sources: src/scrapers/rent/scrape-realestate-au.ts:18-63, src/scrapers/rent/RentScraperService.ts:57-65
Normalization Logic
Normalization is critical for cross-portal comparison. The normalizeListing function converts varied raw data into a consistent format suitable for the rental_listings table.
Key Normalization Components
- Currency Conversion: Uses
loadRatesToEurto fetch exchange rates andbuildLocalConverterto transform original listing prices into the target country's currency. - Room-to-Bedroom Mapping: Converts local room descriptions (e.g., "2-кімнатна") to a standard bedroom count (Rooms - 1, where Studio = 0).
- Periodic Adjustment: Automatically converts Australian weekly rent prices to monthly values for consistency with other regions.
- ID Extraction: Parsers extract the unique source listing ID from URLs or redirect links for deduplication.
Sources: src/scrapers/rent/normalize.ts:1-120, src/scrapers/rent/RentScraperService.ts:48-55
Data Persistence and Schema
The system uses two primary tables to store rental information, separating raw data points from computed statistics.
Database Schema Table
| Table | Column | Type | Description |
|---|---|---|---|
| rental_listings | source_listing_id | TEXT | Unique ID from the source portal |
| bedrooms | INTEGER | Standardized bedroom count | |
| price_local | NUMERIC | Price in the country's main currency | |
| scraped_on | DATE | Used for same-day idempotency | |
| rental_stats | median | NUMERIC | The calculated median price |
| n_listings | INTEGER | Sample size for the statistical bucket | |
| period_end | DATE | End date of the 30-day window |
Sources: src/database/migrations/015_create_rental_tables.sql:5-46
Idempotency and Deduplication
A unique index uq_rental_listings_dedup on (source, source_listing_id, scraped_on) ensures that re-running a scrape on the same day does not create duplicate entries.
Sources: src/database/migrations/015_create_rental_tables.sql:27-29
Aggregation and Statistics
The aggregation phase runs via scripts/aggregate-rent.ts. It processes listings within a trailing 30-day window to generate representative cost data.
Aggregation Process
- Deduplication: Fetches listings using
DISTINCT ON (source, source_listing_id)ordered byscraped_at DESCto ensure only the latest version of a listing in the window is counted. - Bucketing: Listings are grouped by the number of bedrooms.
- Median Calculation: The system calculates the median price for each bedroom bucket.
- Upsert: Results are stored in
rental_stats, keyed by country, city, bedroom count, and period end date.
Sources: src/repositories/RentalListingRepository.ts:39-55, scripts/aggregate-rent.ts:25-50, src/scrapers/rent/aggregate.ts
// Example of SQL used for deduped aggregation window
const sql = `SELECT bedrooms, sqm, price_local
FROM (
SELECT DISTINCT ON (source, source_listing_id)
bedrooms, sqm, price_local, scraped_at
FROM rental_listings
WHERE country_id = $1
AND scraped_at >= NOW() - ($2 || ' days')::interval
ORDER BY source, source_listing_id, scraped_at DESC
) deduped`;
Sources: src/repositories/RentalListingRepository.ts:44-54
Summary
The Rental Property Scraping module provides a robust pipeline for tracking global housing costs. By combining automated browser extraction with strict normalization of currencies and room counts, it transforms disparate real estate listings into comparable statistical data. The use of a 30-day aggregation window and unique indexing ensures that the resulting medians are both accurate and resilient to scraping repetitions.