Terraform Infrastructure as Code

Relevant source files

The following files were used as context for generating this wiki page:

• terraform/main.tf
• terraform/variables.tf
• scripts/deploy-ecr.sh
• scripts/run-ecs-task.sh
• scripts/stop-ecs-task.sh
• package.json

Introduction

The infrastructure for the "WhereIsLifeCheaper" project is managed using Terraform to provide a consistent, version-controlled environment on Amazon Web Services (AWS). This Infrastructure as Code (IaC) setup primarily supports the automated scraping system, which extracts grocery price data from various international supermarkets. The infrastructure is designed around containerized tasks running on AWS Fargate, ensuring scalability and isolation without the overhead of managing underlying EC2 instances.

The scope of the Terraform configuration includes the provisioning of container registries (ECR), orchestration clusters (ECS), secure secret management (AWS Secrets Manager), and automated scheduling (EventBridge). This architecture ensures that scrapers run reliably on a daily schedule, with all necessary credentials and database connection strings securely injected at runtime.

Sources: terraform/main.tf:1-15, terraform/variables.tf:1-15

Core Infrastructure Components

Container Management

The system utilizes AWS Elastic Container Registry (ECR) to store the scraper Docker images. The configuration includes a lifecycle policy to maintain only the five most recent images, preventing storage bloat while allowing for quick rollbacks.

Sources: terraform/main.tf:26-59

Compute and Orchestration

The primary compute resource is an AWS ECS Cluster using the Fargate launch type. Scraper tasks are defined with specific CPU and memory allocations, defaulting to 2 vCPUs and 4GB of RAM to handle resource-intensive browser automation tasks like Playwright.

Sources: terraform/main.tf:62-72, terraform/variables.tf:25-36

Network and Security

The infrastructure leverages the default VPC and its public subnets. A dedicated Security Group is provisioned for ECS tasks, allowing all outbound traffic to facilitate web scraping and database connections, while restricting all unsolicited inbound traffic.

Sources: terraform/main.tf:152-179

Provisioned Resources Summary

Resource Type	Name/ID	Purpose
ECR Repository	whereislifecheaper-scraper	Stores Docker images for the scraping engine.
ECS Cluster	whereislifecheaper	Orchestrates Fargate task execution.
Secrets Manager	database-url	Securely stores the PostgreSQL connection string.
CloudWatch Logs	/ecs/whereislifecheaper/scraper	Centralized logging with 14-day retention.
EventBridge Rule	daily-scrape	Triggers the scraping task daily at 03:00 UTC.

Sources: terraform/main.tf:26-258, terraform/variables.tf:1-45

Architecture and Data Flow

The infrastructure establishes a clear flow from image deployment to scheduled execution. The following diagram illustrates the relationship between the provisioned AWS resources and the deployment scripts.

Task Execution and Permissions

Identity and Access Management (IAM)

Terraform defines several IAM roles to ensure the principle of least privilege:

• Task Execution Role: Used by the ECS agent to pull images from ECR and fetch secrets from Secrets Manager.
• Task Role: Used by the running container to access AWS services at runtime.
• EventBridge Role: Grants permission to EventBridge to invoke ECS RunTask and pass the required IAM roles.

Sources: terraform/main.tf:95-150, terraform/main.tf:216-249

Container Configuration

The ECS Task Definition injects several environment variables and secrets into the scraper container.

Variable Name	Source	Purpose
DATABASE_URL	Secrets Manager	PostgreSQL connection string.
NODE_ENV	Static Value	Set to production.
PLAYWRIGHT_HEADLESS	Static Value	Set to false for browser compatibility.
SCRAPER_PROXY_CONFIG	Secrets Manager	Optional JSON for supermarket-specific proxies.

Sources: terraform/main.tf:181-231

Deployment Scripts and Lifecycle

The project provides shell scripts that interact with the Terraform-managed infrastructure to facilitate CI/CD operations.

Image Deployment

The scripts/deploy-ecr.sh script retrieves the ECR repository URL directly from Terraform outputs, builds the local Docker image, and pushes it to AWS.

# Example deployment flow from the script
ECR_REPO=$(terraform -chdir="../terraform" output -raw ecr_repository_url)
docker build -t whereislifecheaper-scraper .
docker push $ECR_REPO:latest

Sources: scripts/deploy-ecr.sh:1-25

Task Management

• run-ecs-task.sh: Manually triggers a scraper run on Fargate. It can target specific supermarkets by overriding the container command.
• stop-ecs-task.sh: Locates and stops all running tasks within the ECS cluster.

Sources: scripts/run-ecs-task.sh:15-52, scripts/stop-ecs-task.sh:4-22

Configuration Variables

Terraform behavior is customized through variables defined in variables.tf.

Variable	Default Value	Description
aws_region	eu-central-1	Target AWS region.
project_name	whereislifecheaper	Prefix for resource naming.
task_cpu	2048	CPU units (2048 = 2 vCPU).
task_memory	4096	Memory in MB (4 GB).

Sources: terraform/variables.tf:1-36

Conclusion

The Terraform infrastructure provides a robust and automated foundation for the "WhereIsLifeCheaper" scraping engine. By utilizing AWS Fargate and EventBridge, the system achieves serverless, scheduled execution that is both secure and maintainable. The tight integration between Terraform outputs and deployment scripts allows developers to manage the entire lifecycle of the scrapers—from infrastructure provisioning to container deployment—with minimal manual intervention.

Sources: terraform/main.tf:1-275, package.json:20-22