Monitoring & Observability
Relevant source files
The following files were used as context for generating this wiki page:
Monitoring and Observability in the Aether-X Ultimate platform is a multi-layered system designed to provide deep insights into service health, performance metrics, and security audits. The architecture leverages industry-standard tools like Prometheus for metrics collection, Grafana for visualization, and the ELK stack (Elasticsearch, Kibana) for log analysis, ensuring high availability and reliability across the microservices ecosystem.
The observability stack is integrated directly into the infrastructure, providing real-time data on resource utilization, request latency, and error rates. This allows developers and operators to detect anomalies, perform root-cause analysis, and maintain the platform's unrestricted AI operations efficiently.
Sources: README.md:32-34, IMPLEMENTATION_COMPLETE.md:104-109
Infrastructure Components
The monitoring infrastructure is composed of several dedicated services that aggregate data from individual microservices and the API Gateway.
Metrics & Visualization
- Prometheus: Acts as the primary time-series database for collecting metrics. It is configured with a 200-hour retention period and a web interface on port 9090.
- Grafana: Provides the visualization layer, pre-configured with data sources and dashboards to monitor service metrics and alerts. It is accessible on port 3001.
Logging & Analysis
- Elasticsearch: Serves as the search and analytics engine for log data and analytics.
- Kibana: Used for visualizing and navigating Elasticsearch data, specifically for log analysis. Accessible on port 5601.
Sources: docker-compose.yml:181-224, infrastructure/databases/docker-compose.yml:141-180
Monitoring Architecture Diagram
The following diagram illustrates the flow of observability data from the microservices to the centralized monitoring tools.
Health Monitoring & Probes
Every service within the Aether-X Ultimate ecosystem implements a /health endpoint. This is utilized by both Docker Compose and Kubernetes for automated health checks and self-healing.
Health Check Implementation
The standard health response returns a JSON object indicating the service status and identity.
@app.route('/health', methods=['GET'])
def health():
return jsonify({"status": "healthy", "service": "api-gateway"})
Sources: services/api-gateway/app.py:53-55, services/security/app.py:91-93
Kubernetes Probes
In production environments, Kubernetes uses livenessProbes and readinessProbes to manage pod lifecycles.
| Probe Type | Purpose | Configuration Example |
|---|---|---|
| Liveness | Checks if the container is running; restarts if failed. | initialDelaySeconds: 30, periodSeconds: 10 |
| Readiness | Checks if the container is ready to accept traffic. | initialDelaySeconds: 5, periodSeconds: 5 |
Sources: k8s/deployment.yaml:26-44, k8s/api-gateway.yaml:45-56
API Gateway Observability
The API Gateway plays a critical role in observability by intercepting all external traffic and providing centralized logging and rate-limit monitoring.
Key Observability Features
- Request Logging: Captures HTTP method, path, client IP, status code, and duration for every request.
- Rate Limit Tracking: Monitors active clients and total request counts per client IP.
- Metrics Endpoint: A dedicated
/api/v1/metricsendpoint provides a snapshot of gateway performance.
| Endpoint | Method | Description |
|---|---|---|
/health | GET | Returns the operational status of the gateway. |
/api/v1/metrics | GET | Returns active client count, total requests, and service maps. |
Sources: services/api-gateway/app.py:40-51, services/api-gateway/app.py:228-243
Security Auditing
The security service maintains an internal log of sensitive events to ensure auditability of the platform's unrestricted operations.
Security Event Types
The system tracks several specific event types:
LOGIN_ATTEMPT/SUCCESS/FAILUREUNAUTHORIZED_ACCESSENCRYPTION_OPERATIONCUSTOM_PROTOCOL_USAGE
These events include timestamps, user IDs, source IPs, and specific metadata (e.g., encryption data length).
Sources: services/security/app.py:28-48
Testing & Validation
The platform includes a test suite (test.sh) that validates the observability and health of the system through automated checks.
Test Categories
- Health Checks: Iterates through all services to verify the
/healthendpoint responds with a 200 OK status. - Load Testing: Sends a burst of requests (default 100) to the API Gateway to measure success rates and request per second (req/s) throughput.
- Integration Monitoring: Verifies that protected endpoints properly log unauthorized access attempts.
Sources: test.sh:35-51, test.sh:197-227
Summary
Monitoring and Observability in Aether-X Ultimate provides a comprehensive view of the microservices lifecycle. By combining proactive health probes, centralized request logging at the Gateway, and a robust metrics collection stack (Prometheus/Grafana), the platform ensures that its unrestricted AI capabilities remain performant and secure across both development and production (Kubernetes) environments.
Sources: IMPLEMENTATION_COMPLETE.md:104-116, setup-dev.sh:158-162