Telemetry & Observability
Relevant source files
The following files were used as context for generating this wiki page:
The Telemetry & Observability system in FixOps is a distributed architecture designed to monitor, track, and analyze security decision-making processes, API performance, and infrastructure health across cloud and edge environments. It provides deep visibility into the "hot path" of security pipelines, ensuring that decision latencies remain within enterprise-grade guarantees while maintaining a complete evidence trail for compliance audits.
The system is composed of localized telemetry bridges for major cloud providers (AWS, Azure, GCP), edge collectors for distributed ingestion, and centralized dashboards that visualize real-time metrics such as consensus rates, decision latencies, and audit compliance scores. This observability framework allows operators to detect "drift" in security postures and verify the effectiveness of the Decision Engine in both demo and production modes.
Sources: frontend/src/pages/ArchitectDashboard.jsx, tests/e2e/comprehensive_e2e_test.py:1-35, telemetry_bridge/edge_collector/collector_api/app.py
Architecture and Multi-Cloud Ingestion
The telemetry architecture utilizes a bridge pattern to normalize observability data from heterogeneous environments. Each cloud-specific handler (Lambda, Azure Functions, GCP Functions) acts as a specialized ingress point that forwards telemetry events to a centralized processing layer.
Cloud Telemetry Bridges
The system implements specialized handlers for different serverless environments to capture localized execution data:
- AWS Lambda Bridge: Utilizes a Python-based handler to process events and context within the AWS ecosystem.
- Azure Function Bridge: Implements an initialization-based trigger system for capturing Azure-native metrics.
- GCP Function Bridge: Provides a main entry point for Google Cloud Platform telemetry ingestion.
Edge Collector API
An Edge Collector provides a FastAPI-based endpoint for localized telemetry gathering, allowing distributed agents to report status and metrics without direct access to the core database.
The diagram above illustrates the flow of telemetry data from various cloud environments through an edge collector into the core evidence and analytics layers.
Sources: telemetry_bridge/aws_lambda/handler.py, telemetry_bridge/azure_function/init.py, telemetry_bridge/gcp_function/main.py, telemetry_bridge/edge_collector/collector_api/app.py
Key Performance Indicators (KPIs)
The system tracks specific metrics to ensure the Decision Engine meets enterprise performance requirements. These metrics are exposed via the /api/v1/decisions/metrics endpoint and visualized in the Architect Dashboard.
| Metric | Target Value | Description |
|---|---|---|
| Decision Latency | < 300μs (Hot Path) | The time taken to reach a security decision from normalized input. |
| Consensus Rate | ~87% - 95% | The percentage of findings where multiple LLMs or algorithms agree. |
| Audit Compliance | 1.0 (100%) | Verification that every decision is backed by an immutable evidence trail. |
| Noise Reduction | > 70% | The reduction in false positives compared to traditional SAST/SCA tools. |
Sources: frontend/src/pages/EnhancedDashboard.jsx, frontend/src/pages/ArchitectDashboard.jsx, tests/e2e/comprehensive_e2e_test.py
Decision Observability and LLM Comparison
FixOps provides specialized observability for AI-driven decisions through the "Enhanced Dashboard." This system monitors the variance and disagreement between different AI models (GPT-4, Claude, Gemini) to calculate a "Consensus Confidence" score.
Disagreement Analysis
When models disagree, the observability system triggers an "Expert Review" requirement. The system tracks:
- Confidence Variance: The spread of confidence levels across different models.
- Decision Split: Whether models reached conflicting verdicts (e.g., ALLOW vs. BLOCK).
- Areas of Disagreement: Specific categories such as risk assessment, exploitability, or business impact where models diverged.
The sequence shows how the telemetry system intercepts multi-LLM outputs to monitor variance and trigger human intervention when AI confidence is low.
Sources: frontend/src/pages/EnhancedDashboard.jsx
System Health and Validation
The observability framework includes comprehensive health checking and consistency validation, primarily managed through the E2E test suite and system status endpoints.
Health Endpoints
The platform exposes several levels of health data:
/health: Basic service availability check./api/v1/status: Detailed component status including the availability of the processing layer./api/v1/system/metrics: Low-level system resource utilization./api/v1/feeds/health: Tracking the availability and freshness of vulnerability intelligence feeds (NVD, KEV, EPSS).
Consistency Checks
Automated observability tasks verify that data ingested via the pipeline is correctly reflected across all endpoints. This includes:
- Inventory Consistency: Verifying registered applications appear in the asset inventory.
- Findings Consistency: Ensuring SARIF/SBOM findings are accurately counted in analytics.
- Pipeline Traceability: Confirming that pipeline runs process the exact number of components reported during ingestion.
Sources: tests/e2e/comprehensive_e2e_test.py:1150-1250, frontend/src/pages/ArchitectDashboard.jsx
Conclusion
Telemetry & Observability in FixOps serves as the backbone for trust in automated security decisions. By combining multi-cloud ingestion bridges with advanced disagreement analysis and strict consistency checking, the system provides the transparency required for enterprise DevSecOps deployments. Whether running in a lightweight demo mode or a high-performance production environment, the observability layer ensures that every decision is traceable, performant, and verified against the established security baseline.