Rent Backend Service

Relevant source files

The following files were used as context for generating this wiki page:

• src/scrapers/rent/RentScraperService.ts
• src/repositories/RentalListingRepository.ts
• scripts/aggregate-rent.ts
• src/scrapers/rent/normalize.ts (Derived from test logic)
• src/scrapers/rent/types.ts (Referenced in service imports)
• frontend/src/services/api.ts
• scripts/scrape-rent.ts

The Rent Backend Service is a specialized module within the WhereIsLifeCheaper project designed to ingest, normalize, and aggregate rental market data from various international sources. Its primary purpose is to provide cost-of-living comparisons specifically focused on housing, complementing the project's core grocery price comparison functionality. Sources: README.md, CLAUDE.md, src/scrapers/rent/RentScraperService.ts:1-15

The service operates in two main phases: an ingestion phase where raw data is scraped from regional portals (such as OLX, Domria, and Domain.au) and a processing phase where these listings are deduplicated and aggregated into statistical medians by bedroom count. Sources: src/scrapers/rent/RentScraperService.ts:39-45, scripts/aggregate-rent.ts:25-30

Architecture and Data Flow

The architecture follows a pipeline pattern consisting of scrapers, a normalization layer, and a repository for persistence. The flow starts with scheduled scripts that trigger the RentScraperService, which orchestrates various source-specific scrapers. Sources: src/scrapers/rent/RentScraperService.ts:40-42, scripts/scrape-rent.ts

Ingestion Flow

The ingestion process iterates through RENT_TARGETS, which define specific country/city pairs. For each target, the service retrieves relevant scrapers mapped to that country code. Sources: src/scrapers/rent/RentScraperService.ts:47-53

The diagram shows the sequence of operations from script execution to database persistence. Sources: src/scrapers/rent/RentScraperService.ts:40-75, scripts/scrape-rent.ts:5-8

Data Normalization

Normalization is critical for cross-source comparison. The service converts varied room descriptions (e.g., "2-кімнатна" vs "1 Bed") into a standardized bedrooms count and translates various currencies into the target country's local currency using exchange rates. Sources: src/scrapers/rent/tests/normalize.test.ts:10-35

Key normalization logic includes:

• Room Conversion: Translating "X-room" formats to "X-1" bedrooms for specific regions like Ukraine, or direct mapping for others. Sources: src/scrapers/rent/tests/normalize.test.ts:13-18
• Currency Conversion: Converting all original prices (USD, EUR, etc.) to the local currency (e.g., UAH or AUD) to ensure statistical consistency within a city. Sources: src/scrapers/rent/RentScraperService.ts:55-58
• Period Adjustment: Normalizing weekly rent (common in Australia) to monthly costs. Sources: src/scrapers/rent/tests/normalize.test.ts:121-125

Repository and Storage

The RentalListingRepository manages interactions with the PostgreSQL database. It uses a batch insertion strategy with conflict handling to avoid duplicate entries for the same listing on the same day. Sources: src/repositories/RentalListingRepository.ts:15-30

Database Schema (Rental Listings)

Column	Type	Description
country_id	Integer	Link to countries table
city	String	Target city name
source	String	Scraper source (olx, domria, etc.)
source_listing_id	String	Unique ID from the portal
bedrooms	Integer	Standardized bedroom count
sqm	Numeric	Square meters (nullable)
price_local	Numeric	Normalized price in country currency
scraped_on	Date	Partition/Index key for deduplication

Sources: src/repositories/RentalListingRepository.ts:8-13, src/repositories/tests/RentalListingRepository.test.ts:25-30

Aggregation Logic

Aggregation occurs via the scripts/aggregate-rent.ts utility. This script calculates the median price for different housing sizes over a 30-day window. It employs a DISTINCT ON query to ensure that if a listing was scraped multiple times within the window, only the most recent version is considered. Sources: scripts/aggregate-rent.ts:18-35, src/repositories/RentalListingRepository.ts:42-50

The sequence illustrates how raw listings are transformed into statistical summaries. Sources: scripts/aggregate-rent.ts:25-50

API Integration

The frontend accesses this data through the rentApi in frontend/src/services/api.ts. The primary endpoint GET /api/rent returns the aggregated statistics for each country, which are then displayed in the RentComparison component. Sources: frontend/src/services/api.ts:133-138, frontend/src/components/comparison/RentComparison.tsx:18-22

Endpoint Summary

Method	Endpoint	Description
GET	/api/rent	Returns CountryRent[] containing aggregated buckets per country.

Sources: frontend/src/services/api.ts:134

Summary

The Rent Backend Service provides a robust pipeline for monitoring international rental markets. By decoupling raw scraping from statistical aggregation, the system maintains a historical record of individual listings in rental_listings while providing high-performance, pre-calculated medians for the user dashboard via the aggregation layer. Sources: scripts/aggregate-rent.ts:10-15, src/scrapers/rent/RentScraperService.ts:39-42