Agent Orchestration
Relevant source files
The following files were used as context for generating this wiki page:
Agent Orchestration in the ClassMate project refers to the systematic coordination of multiple specialized AI agents to process lecture transcripts and generate structured educational content. The system employs a multi-agent architecture where an orchestrator manages task routing, execution planning, and quality evaluation to transform raw text into summaries, key points, and study materials.
The orchestration layer acts as the bridge between raw transcription data and the final user-facing notes, ensuring that complex processing tasks are broken down into manageable steps executed by specialized agents. Sources: ai_backend/agents/orchestrator.py:11-20, ai_workers/notes_worker.py:40-50
Core Architecture and Orchestration Logic
The system utilizes a NotesOrchestrator to manage a suite of specialized agents. This orchestrator is responsible for initializing the agents, determining the execution plan based on user requirements or automated routing, and ensuring the final output meets quality standards through a refinement loop.
Orchestration Components
| Component | Class / Agent | Purpose |
|---|---|---|
| Orchestrator | NotesOrchestrator | Coordinates the lifecycle and execution flow of all agents. |
| Router | RouterAgent | Analyzes transcripts to provide planning hints and influence execution. |
| Summarizer | SummaryAgent | Generates concise overviews of the transcript content. |
| Point Extractor | KeyPointsAgent | Identifies critical concepts and highlights from the text. |
| Task Extractor | ActionItemsAgent | Extracts specific tasks or requirements mentioned in the lecture. |
| Quality Control | EvaluatorAgent | Scans outputs for issues like empty summaries or excessive list lengths. |
| Correction | RefinerAgent | Corrects issues identified by the Evaluator, such as trimming lists. |
Sources: ai_backend/agents/orchestrator.py:11-20, ai_backend/agents/note_agents.py:11-135
Execution Flow
The orchestration process follows a structured sequence where the outcome of one agent often serves as the context for another.
The execution plan is dynamic. While explicit user flags (e.g., include_summary) take priority, the RouterAgent provides hints that can influence which agents are activated. Sources: ai_backend/agents/orchestrator.py:22-45
Specialized Agent Modules
1. Note Generation Agents
These agents perform the primary transformation of transcript data. The SummaryAgent uses a heuristic approach (splitting text into sentences) while KeyPointsAgent and ActionItemsAgent use keyword-based filtering to identify relevant sentences.
Sources: ai_backend/agents/note_agents.py:11-70
2. Post-Transcription Chain
Before orchestration for notes begins, a PostTranscriptionChain cleans the raw transcript. This involves specialized sub-agents:
- CleanupAgent: Normalizes whitespace and punctuation spacing.
- TermConsistencyAgent: Replaces inconsistent terms based on learned mappings.
- ConfidenceFilterAgent: Flags segments with low Whisper confidence scores.
Sources: ai_backend/agents/transcription_post_agents.py:84-118
3. Study Assistant Integration
Beyond standard notes, the StudyAssistantAgent provides a LangChain-powered interface for advanced interactions. It uses a VectorMemoryStore to index content for semantic search, allowing it to generate flashcards, quizzes, and answer user questions based on lecture context.
Sources: ai_backend/agents/study_assistant.py:48-75, ai_backend/api/ai_agent_api.py:32-60
Data Management and Memory
The orchestration system relies on an AgentMemory system to maintain state across different agent runs. This memory captures "events" (e.g., summary_generated, evaluation_done) which are included in the final snapshot.
Agent Memory Structure
Worker and API Integration
The orchestration logic is exposed via two primary channels:
- Celery Worker: The
NotesWorkerwraps theNotesOrchestratorfor background processing. It updates the database with the resulting summary and key points once generation is complete. Sources: ai_workers/notes_worker.py:33-65 - FastAPI AI Agent API: Provides synchronous endpoints for specific features like
/flashcardsor/quizthat utilize theStudyAssistantAgent. Sources: ai_backend/api/ai_agent_api.py:53-85
API Request Schema (Study Guide)
{
"session_id": "string",
"transcript": "string"
}
Sources: ai_backend/api/ai_agent_api.py:42-44
Conclusion
Agent Orchestration in ClassMate provides a modular and extensible framework for educational content generation. By separating concerns into distinct agents—summarization, extraction, evaluation, and refinement—the system maintains high output quality while allowing for the integration of both simple heuristic-based processing and advanced RAG (Retrieval-Augmented Generation) workflows via LangChain.