Celery and Redis Background Jobs
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Background job processing in this project is handled by Celery, using Redis as the message broker and result backend. This system is designed to handle long-running operations—such as transaction enrichment, AI-powered categorization, and large CSV data imports—without blocking the main API response or user interface. Sources: README.md:16-19, CLAUDE.md:120-123
The architecture separates the application into a FastAPI backend (producer) and Celery workers (consumers). It also includes a Celery Beat service to handle scheduled/periodic tasks, such as daily bank synchronization and data refreshes. Sources: README.md:120-130, deploy/compose/docker-compose.yml:102-124
Architecture and Components
The background job system follows a standard producer-consumer architecture, with Redis acting as the communication bridge.
The diagram above shows the relationship between the API producers, the Redis broker, and the Celery workers that process tasks and update the shared database. Sources: CLAUDE.md:14-25, deploy/compose/docker-compose.yml:1-124
Key Components
| Component | Purpose | Configuration Details |
|---|---|---|
| Redis | Message broker and result backend | Uses redis:7-alpine. Defaults to port 6379. |
| Celery Worker | Executes background tasks | Processes tasks from the queue; supports concurrency (default: 4). |
| Celery Beat | Periodic task scheduler | Dispatches tasks at specified intervals or cron-like schedules. |
| Broker/Backend | Task distribution and results | Both use REDIS_URL (e.g., redis://localhost:6379/0). |
Sources: backend/celery_app.py:8-14, docker-compose.yml:43-83, deploy/compose/docker-compose.yml:102-124
Configuration and Implementation
The Celery application is configured in backend/celery_app.py. It initializes a Celery instance named finance_tasks and sets various performance and reliability parameters. Sources: backend/celery_app.py:10-14
Celery App Settings
The application is configured with strict serialization and time limits to ensure stability:
- Serializers: Uses
jsonfor task data and results. - Time Limits: Hard limit of 3600s (1 hour) and soft limit of 3300s (55 minutes).
- Concurrency: Set to 4 workers by default in the execution environment.
- Prefetch Multiplier: Set to
1to prevent a single worker from hogging tasks, ensuring better distribution across multiple workers.
Sources: backend/celery_app.py:18-40, deploy/compose/docker-compose.yml:102-105
Implementation Example: Background CSV Import
The frontend can enqueue a background CSV import via the enqueueBackgroundImport server action. This communicates with the FastAPI backend endpoint /api/csv-import/enqueue, which then delegates the long-running processing to the Celery worker. Sources: frontend/lib/actions/csv-import.ts:881-900
The sequence above illustrates the asynchronous flow of a CSV import, allowing the user to continue using the application while data is processed. Sources: frontend/lib/actions/csv-import.ts:881-987
Task Discovery and Management
The system automatically discovers tasks located in the tasks directory of the backend. Specifically, it is configured to include tasks.csv_import_tasks for handling transaction imports. Sources: backend/celery_app.py:15, backend/celery_app.py:44
Deployment Commands
Individual services are run as separate containers in production environments:
- Worker:
celery -A celery_app worker --loglevel=info --concurrency=4 - Beat:
celery -A celery_app beat --loglevel=info
Sources: deploy/railway/README.md:46-48, deploy/compose/docker-compose.yml:102-124
Environment Variables
Both Celery and Redis rely on specific environment variables for connectivity:
| Variable | Description |
|---|---|
REDIS_URL | Redis connection string (e.g., redis://redis:6379/0) |
DATABASE_URL | Used by workers to update state in the PostgreSQL database |
OPENAI_API_KEY | Optional; used by workers for AI-powered transaction enrichment |
Sources: backend/celery_app.py:8, deploy/compose/docker-compose.yml:106-114, CLAUDE.md:120-123
Summary
Celery and Redis provide the asynchronous backbone for the Personal Finance App, enabling high-performance features like AI transaction enrichment and bulk data synchronization. By offloading these tasks from the primary API, the system maintains a responsive user experience while ensuring data integrity through the shared PostgreSQL database. Sources: README.md:16-19, CLAUDE.md:14-25