Cloud Storage Integration
Relevant source files
The following files were used as context for generating this wiki page:
The Cloud Storage Integration in ClassMate provides a unified interface for persisting large binary objects, specifically audio recordings and generated transcripts. This system is designed for high scalability and multi-provider compatibility, ensuring that lecture data is stored securely and remains accessible across various platforms like the mobile app and web dashboard.
The integration serves as the primary data persistence layer for the Recording Pipeline and the AI Workers. It abstracts the complexities of different cloud providers (AWS, GCP, Azure) and local development environments (MinIO), allowing the backend to handle file operations through a consistent service class.
Sources: ai_backend/services/cloud_storage_service.py:1-15, PROJECT_STRUCTURE.md:120-130, README.md:18-35
System Architecture
The cloud storage architecture follows a service-oriented approach. A central CloudStorageService class manages initialization and operations based on environment variables. In production, this typically interfaces with AWS S3, while local development utilizes MinIO to simulate object storage.
Multi-Provider Support
The system is architected to support three major cloud providers and an S3-compatible local provider. The choice of provider is determined by the CLOUD_STORAGE_PROVIDER environment variable.
| Provider | Backend Client | Target Resource |
|---|---|---|
| AWS | boto3 | S3 Buckets |
| GCP | google-cloud-storage | GCS Buckets |
| Azure | azure-storage-blob | Blob Containers |
| Local | boto3 (S3 API) | MinIO Containers |
Sources: ai_backend/services/cloud_storage_service.py:19-30, infra/README.md:25-35, docker-compose.prod.yml:58-75
Infrastructure Layout
The infrastructure is provisioned using Terraform (for AWS) or defined in Docker Compose (for local/production on-premise).
The diagram shows how the CloudStorageService acts as a gateway to multiple physical storage backends.
Sources: infra/terraform/main.tf:190-215, ai_backend/services/cloud_storage_service.py:32-65
Core Service Logic
The CloudStorageService handles three primary types of operations: uploading audio files, uploading transcript JSONs, and managing file metadata.
Audio Upload Workflow
Audio files are stored using a hierarchical path structure: audio/{user_id}/{session_id}/{file_name}. The service generates pre-signed URLs to allow controlled access for both uploads (PUT) and downloads (GET).
The sequence illustrates the generation of secure URLs for session-based storage. Note that signed URLs are currently configured with a 7-day expiry for downloads.
Sources: ai_backend/services/cloud_storage_service.py:67-112, MISSING_INFO.md:73-77
File Management Functions
The service provides methods to list, delete, and query storage usage per user.
list_user_files(user_id, file_type): Filters objects based on prefixes likeaudio/ortranscripts/.delete_file(file_name): Removes objects from the configured bucket.get_storage_usage(user_id): Aggregates the size and count of audio and transcript files to provide usage statistics.
Sources: ai_backend/services/cloud_storage_service.py:186-210, ai_backend/services/cloud_storage_service.py:280-305
Configuration and Environment
Storage behavior is strictly controlled through environment variables. The system defaults to AWS if no provider is specified.
Configuration Parameters
| Variable | Description | Default/Example |
|---|---|---|
CLOUD_STORAGE_PROVIDER | Defines which driver to initialize | aws, gcp, azure |
CLOUD_STORAGE_BUCKET | The name of the storage container | classmate-recordings |
MINIO_ENDPOINT | Local S3-compatible endpoint | minio:9000 |
AWS_ACCESS_KEY_ID | Credentials for AWS/MinIO | Found in .env |
S3_BUCKET_NAME | S3-specific bucket name | classmate-audio-storage |
Sources: ai_backend/services/cloud_storage_service.py:19-25, infra/README.md:38-45, MISSING_INFO.md:41-45
Terraform Resource Definition
For AWS deployments, the storage bucket is provisioned with versioning and server-side encryption enabled.
resource "aws_s3_bucket" "classmate" {
bucket = "classmate-audio-storage-${random_id.bucket_suffix}"
tags = {
Name = "classmate-audio-storage"
Environment = "production"
}
}
resource "aws_s3_bucket_versioning" "classmate" {
bucket = aws_s3_bucket.classmate.id
versioning_configuration {
status = "Enabled"
}
}
Sources: infra/terraform/main.tf:190-205
Implementation Status and Gaps
While the storage service is functionally complete for basic operations, several security and integration gaps have been identified for production readiness.
- URL Expiry: Current signed URLs expire in 7 days. Recommendations suggest reducing this to 1-4 hours and binding them to user sessions for better security.
- Upload Endpoint: The mobile app's
upload_queue_service.dartrequires a manual update to point to the actual backend URL instead of the placeholderapi.classmate.app/upload. - GDPR Compliance: Deletion currently only purges database rows; the storage service must be explicitly called to remove the corresponding files from the cloud provider.
- Local Storage Fallback: The backend logs a warning if SQLite is used instead of PostgreSQL, as the current storage logic assumes a multi-worker environment that can access shared cloud resources.
Sources: MISSING_INFO.md:46-51, MISSING_INFO.md:73-82, GAPS_IMPLEMENTATION_SUMMARY.md:410-425
Conclusion
Cloud Storage Integration is a critical component of ClassMate that enables the persistent storage of meeting data. By using a provider-agnostic service layer and infrastructure-as-code, the system maintains flexibility across development and production environments. Future enhancements will focus on tightening the security of signed URLs and ensuring full compliance with data deletion regulations.