NoSQL & Content Storage
Relevant source files
The following files were used as context for generating this wiki page:
The Aether-X Ultimate platform employs a multi-layered database strategy to handle diverse data requirements, ranging from structured user management to high-volume unstructured AI content storage. The NoSQL layer is specifically designed to manage generated content, unstructured application data, and high-availability storage needs that are not suited for traditional relational databases.
This storage architecture supports the platform's core services, including text, image, and video generation, by providing scalable backends for persisting metadata and binary assets. The system integrates MongoDB for document-based content storage, Cassandra for high-availability distributed data, and Elasticsearch for full-text search and analytics capabilities.
Sources: README.md:37-43, IMPLEMENTATION_COMPLETE.md:16-25
Architecture Overview
The NoSQL and content storage infrastructure is containerized and orchestrated via Docker Compose. Each storage engine serves a specific role within the microservices ecosystem.
Component Breakdown
| Component | Technology | Primary Purpose | Port |
|---|---|---|---|
| Document Store | MongoDB 6.0 | Content storage, unstructured app data, and metadata | 27017 |
| Wide-Column Store | Cassandra 4.1 | High-availability data storage and distributed persistence | 9042 |
| Search Engine | Elasticsearch 8.8.0 | Full-text search, analytics, and log indexing | 9200 |
| Distributed Queue | Apache Kafka | Real-time data streaming and message persistence | 9092 |
Sources: infrastructure/databases/docker-compose.yml:45-134, README.md:73-82
The following diagram illustrates the relationship between the generating services and the storage layer:
The diagram shows how specialized generation services interface with specific NoSQL backends for persistence and processing.
Sources: docker-compose.yml:25-95, infrastructure/databases/docker-compose.yml:45-134
MongoDB: Document and Content Storage
MongoDB acts as the primary repository for unstructured data, specifically for the Image and Video generation services. It utilizes the aether_x_db database to store collections that require flexible schemas.
Schema Validation
MongoDB is configured with JSON schema validation to ensure data integrity while maintaining document flexibility. The primary collections include:
- users: Stores profile information and credentials.
- apps: Stores metadata for no-code applications created by users.
- workflows: Stores JSON-based logic for automated workflows.
- content_generated: The main repository for AI-generated outputs, including content types and binary links.
Sources: infrastructure/databases/init-databases.sh:100-165
Data Model Fields: content_generated
| Field | Type | Description |
|---|---|---|
user_id | objectId | Reference to the owner of the content |
content_type | string | Type of content (e.g., image, video, text) |
content_data | string | The actual payload or reference path |
metadata | object | JSON object containing generation parameters |
created_at | date | Timestamp of generation |
Sources: infrastructure/databases/init-databases.sh:147-158
Cassandra: High-Availability Storage
Cassandra is utilized for high-throughput, distributed storage of generated content and user data, ensuring high availability across the aether_x_keyspace.
Keyspace Configuration
The keyspace is initialized with SimpleStrategy and a replication_factor of 1 for the base implementation, suitable for single-datacenter deployments.
CREATE KEYSPACE IF NOT EXISTS aether_x_keyspace
WITH REPLICATION = {
'class' : 'SimpleStrategy',
'replication_factor' : 1
};
Sources: infrastructure/databases/init-databases.sh:169-174
Table Structures
Cassandra tables mirror the core data entities but optimize for distributed access patterns using UUIDs as primary keys and secondary indexes on high-query fields like username and user_id.
Sources: infrastructure/databases/init-databases.sh:178-216
Elasticsearch & Analytics
Elasticsearch provides the platform's search and analytics backbone, primarily used by the API Gateway and Data Processing services for full-text indexing and real-time monitoring.
Search and Monitoring Integration
- Discovery: Configured as a single-node cluster with security disabled for internal service communication.
- Kibana Integration: A Kibana instance is deployed on port 5601, connected to Elasticsearch for log visualization and data exploration.
- Service Usage: The
text-generationandai-modelsservices depend on Elasticsearch for retrieving indexed content and training data metadata.
Sources: infrastructure/databases/docker-compose.yml:111-155, docker-compose.yml:13-55
This sequence shows the interaction between microservices and the search/analytics stack.
Sources: infrastructure/databases/docker-compose.yml:135-155, docker-compose.yml:45-55
Data Ingestion & Streaming
The data-processing service manages the ingestion of data into the NoSQL ecosystem using Apache Kafka for real-time streaming.
Ingestion Flow
- Ingest API: Data is received via
POST /api/v1/data/ingest. - Queueing: Data is placed into an internal queue and assigned a unique
message_id. - Kafka Streaming: For production workloads, Kafka handles message persistence and distribution to downstream consumers.
- Transformation: Data is normalized or filtered (e.g., removing sensitive information) before final persistence in MongoDB or Elasticsearch.
Sources: services/data-processing/app.py:20-50, services/data-processing/app.py:165-190, infrastructure/databases/docker-compose.yml:85-110
Summary
The NoSQL and Content Storage system in Aether-X Ultimate is a multi-modal architecture designed for scale. By separating concerns between MongoDB (flexible document storage), Cassandra (distributed availability), and Elasticsearch (searchable analytics), the platform ensures that AI-generated content is persisted efficiently and remains highly accessible for both end-users and internal processing pipelines.