Docker & AWS ECS Containerization
Relevant source files
The following files were used as context for generating this wiki page:
The WhereIsLifeCheaper project utilizes Docker and AWS Elastic Container Service (ECS) to provide a scalable and automated environment for its web scraping infrastructure. The system is designed to containerize the scraper service, enabling it to run in a consistent environment that includes the necessary dependencies for Playwright browser automation.
The containerization strategy supports two primary workflows: local development using Docker Compose for PostgreSQL orchestration and production deployment on AWS using ECS Fargate. In production, the scrapers are triggered as scheduled tasks via Amazon EventBridge, allowing for daily automated price collection without the overhead of persistent server management.
Sources: plan.md:12-16, terraform/main.tf:267-272
Infrastructure Architecture
The project's AWS infrastructure is defined using Terraform, focusing on an ECR-to-ECS pipeline. The architecture leverages Amazon ECR for image storage and ECS Fargate for serverless container execution within a VPC.
The diagram illustrates the flow from local code build to automated execution on AWS ECS.
Key Components
| Component | Description |
|---|---|
| ECR Repository | Stores the whereislifecheaper-scraper Docker images with a lifecycle policy keeping the last 5 images. |
| ECS Cluster | Logical grouping of Fargate tasks named via the ecs_cluster_name variable. |
| Task Definition | Configures the scraper container, including CPU (default 2048), Memory (default 4096), and environment variables. |
| Secrets Manager | Securely stores the DATABASE_URL, Google Cloud credentials, and proxy configurations. |
| EventBridge | Triggers the ECS Task daily at 3 AM UTC. |
Sources: terraform/main.tf:28-44, terraform/main.tf:192-205, terraform/variables.tf:22-38, terraform/main.tf:271-274
Container Configuration
The scraper container is configured for production environments with specific settings to optimize Playwright's browser automation.
Environment Variables
The ECS Task Definition injects several environment variables into the scraper container to control its runtime behavior:
- NODE_ENV: Set to
production. - PLAYWRIGHT_HEADLESS: Set to
false(allowing headed execution if needed in specific environments). - SCRAPER_MAX_RETRIES: Configured to
3for handling network or selector failures. - DISABLE_DEV_SHM: Set to
trueto prevent shared memory issues in container environments. - SEED_DB / SYNC_RATES: Triggers database initialization and exchange rate synchronization on startup.
Sources: terraform/main.tf:204-245
Secret Integration
Sensitive information is pulled from AWS Secrets Manager and mapped directly to container environment variables:
| Secret Name | Container Variable |
|---|---|
database-url | DATABASE_URL |
google-credentials | GOOGLE_CREDENTIALS_JSON |
proxy-config | SCRAPER_PROXY_CONFIG |
Sources: terraform/main.tf:248-261
Deployment & CI/CD
The project includes shell scripts and NPM commands to facilitate the transition from code to a running container.
Build and Push Process
The scripts/deploy-ecr.sh script automates the Docker lifecycle for the scraper. It retrieves the repository URL from Terraform outputs, authenticates with AWS ECR, builds the local image, and pushes the latest tag.
# Build and Tag
docker build -t whereislifecheaper-scraper .
docker tag whereislifecheaper-scraper:latest $ECR_REPO:latest
# Push to AWS
aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $ECR_REPO
docker push $ECR_REPO:latest
Sources: scripts/deploy-ecr.sh:13-24, terraform/outputs.tf:46-59
Task Execution Management
Management of the ECS tasks is handled through both automated schedules and manual scripts defined in package.json:
npm run docker:build: Local image creation.npm run aws:deploy: Triggers the ECR deployment script.npm run aws:run: Executes a one-off ECS task usingscripts/run-ecs-task.sh.
Sources: package.json:18-22
Network & Security
The ECS tasks run within the default VPC using a public subnet approach.
The diagram shows the security group configuration allowing only outbound traffic.
The aws_security_group.ecs_tasks resource is configured with no inbound rules, as scrapers only initiate outbound connections for web scraping and database synchronization. All outbound traffic (protocol -1) is permitted to ensure the containers can reach supermarket websites and external databases.
Sources: terraform/main.tf:161-179
Conclusion
The containerization strategy for WhereIsLifeCheaper provides a robust, serverless framework for daily grocery price scraping. By combining Docker's environment consistency with AWS ECS Fargate's scalability and Secrets Manager's security, the system ensures that complex scraping tasks are executed reliably and automatically.