Async Worker System

Relevant source files

The following files were used as context for generating this wiki page:

• ai_workers/notes_worker.py
• PROJECT_STRUCTURE.md
• README.md
• docker-compose.prod.yml
• ai_backend/api/public_api.py

The Async Worker System in ClassMate is a distributed task processing framework designed to handle resource-intensive AI operations outside the main request-response cycle of the FastAPI backend. By leveraging Celery as a task queue and Redis as a message broker, the system ensures that long-running processes such as audio transcription and AI-powered note generation do not block the user interface or API responsiveness.

This system is a core component of the "Data & Scaling" layer, enabling horizontal scalability by allowing multiple worker instances to process tasks from a shared queue. It supports the application's hybrid scalable architecture, designed to handle over 100,000 users by offloading heavy lifting to containerized AI workers.

Sources: README.md:9-25, PROJECT_STRUCTURE.md:144-149, ai_workers/notes_worker.py:23-28

Architecture Overview

The system follows a producer-consumer pattern where the FastAPI backend (producer) dispatches tasks to Redis, and specialized Celery workers (consumers) execute them.

The diagram above illustrates the flow from API request to background execution and eventual database persistence.
Sources: README.md:13-25, docker-compose.prod.yml:102-126, ai_workers/notes_worker.py:23-28

Key Components

Component	Technology	Role
Broker	Redis	Acts as the message transport layer between the backend and workers.
Backend API	FastAPI	Initiates tasks via Celery client when users upload audio or request notes.
Task Queue	Celery	Manages task distribution, retries, and worker concurrency.
Workers	Python/Celery	Specialized containers (e.g., NotesWorker, TranscriptionWorker) for AI processing.
Database	PostgreSQL	Final destination for processed transcripts and generated notes.

Sources: README.md:13-25, PROJECT_STRUCTURE.md:144-149, docker-compose.prod.yml:102-126

Notes Generation Worker

The NotesWorker is a specialized service responsible for transforming raw meeting transcripts into structured study materials. It utilizes a multi-agent orchestrator to generate summaries, key points, and action items.

Task Execution Flow

1. Task Arrival: The generate_notes_task is triggered via Celery.
2. Orchestration: The NotesWorker invokes the NotesOrchestrator to process the transcript.
3. Database Update: The worker executes a SQL UPDATE to store the results in the sessions table.
4. Completion: The task returns a success status or logs failure details.

The sequence diagram shows the internal logic of the notes worker as it interacts with the AI orchestrator and the database.
Sources: ai_workers/notes_worker.py:34-106

Data Structure for Notes

The worker processes and saves the following data structure to the database:

notes = {
    "session_id": session_id,
    "summary": result.get("summary"),
    "key_points": result.get("key_points", []),
    "action_items": result.get("action_items", []),
    "evaluation": result.get("evaluation"),
    "agent_meta": result.get("agent_meta"),
    "memory": result.get("memory"),
}

Sources: ai_workers/notes_worker.py:46-54

Infrastructure & Configuration

The worker system is deployed using Docker Compose for production-grade reliability. Workers are configured with resource limits to prevent memory exhaustion during heavy AI processing.

Docker Configuration

In the production environment, the worker service is defined with specific concurrency and memory limits:

• Command: celery -A ai_backend.celery_client worker --loglevel=info --concurrency=4
• Memory Limit: 2G
• CPU Limit: 2.0
• Dependencies: Relies on healthy postgres and redis services before starting.

Sources: docker-compose.prod.yml:102-126

Environment Variables

Workers require several environment variables for operation:

• DATABASE_URL: Connection string for PostgreSQL (e.g., postgresql://user:pass@postgres:5432/classmate).
• REDIS_URL: Connection string for the Redis broker (e.g., redis://:pass@redis:6379/0).
• OPENAI_API_KEY: Required for LLM-based note generation tasks.

Sources: docker-compose.prod.yml:106-111, ai_workers/notes_worker.py:28

Task Dispatching

Tasks are dispatched from the public API layer. For instance, the transcription endpoint offloads the audio processing to a Celery task.

@router.post("/transcribe")
async def public_transcribe(request: Dict[str, Any], current_user: dict = Depends(get_current_user)):
    # ... validation logic ...
    from ..tasks import process_transcription_task
    task = process_transcription_task.delay(session_id, audio_path)
    return JSONResponse(content={
        "message": "Transcription queued",
        "task_id": task.id,
        "status": "processing"
    })

Sources: ai_backend/api/public_api.py:126-153

Summary

The Async Worker System is a critical architectural component that enables ClassMate to handle computationally expensive AI tasks. By decoupling transcription and note generation from the API, the system provides a scalable foundation for high-concurrency environments. The use of Celery and Redis allows for flexible worker scaling and ensures data integrity through persistent database updates upon task completion.