Tracing & Observability
Relevant source files
The following files were used as context for generating this wiki page:
Tracing and Observability components provide deep visibility into the system's runtime behavior, performance, and security posture. This module tracks requests as they flow through the Architecture layers, monitors financial data processing health, and captures security audit data. The project implements a multi-layered observability strategy including structured logging, real-time performance metrics, error tracking via Sentry, and automated security scanning.
Sources: AUDIT_REPORT.md, IMPROVEMENTS.md:120-135, GEMINI.md:5-15
Request Tracing and Middleware
The tracing system uses middleware to intercept every incoming HTTP request to the FastAPI backend. It assigns unique correlation IDs to requests, allowing developers to follow a single transaction across multiple internal service calls and database operations.
Tracing Flow
The middleware initiates a span at the start of a request and terminates it upon sending the response. If an error occurs, the middleware attaches the stack trace to the span before propagating the error to the global handler.
This diagram illustrates the sequential interception of requests by tracing and metrics middleware to ensure every transaction is timed and logged.
Sources: src/api/middleware/tracing_middleware.py, api/middleware/request_metrics.py, IMPROVEMENTS.md:130-140
Structured Logging
The application utilizes a centralized logger for event recording. While the initial implementation uses basic logging, the architectural mandate requires a transition to structured JSON logging to facilitate ingestion by log management platforms.
Logging Standards
- Format: Logs must use JSON format in production to include metadata like timestamps, log levels, and event types.
- Context: Log entries should include the
execution_idorcorrelation_idwhere applicable. - Verbosity: The system uses
DEBUGfor algorithm execution steps andINFOfor business events like asset relationship discovery.
| Feature | Description | Implementation Status |
|---|---|---|
| JSON Logging | Structured event recording for machine readability | Recommended (IMPROVEMENTS.md) |
| Log Levels | Consistent use of DEBUG, INFO, WARNING, ERROR | Partially Implemented |
| Correlation | Linking logs across service boundaries via IDs | Active in Middleware |
Sources: src/observability/logger.py, IMPROVEMENTS.md:35-50, AUDIT_REPORT.md:65-72
Metrics and Monitoring
The RequestMetrics middleware collects performance data used for system health assessments. It tracks quantitative data points such as request volume, latencies, and error rates.
Collected Metrics
- Request Counter: Tracks total hits to API endpoints categorized by HTTP status codes (2xx, 4xx, 5xx).
- Latency Histograms: Measures the time taken to process asset relationship calculations and visualization data generation.
- Dependency Health: Monitors the response times of external data providers like Yahoo Finance.
The metrics collection flow shows how middleware instruments API calls to feed external monitoring tools.
Sources: api/middleware/request_metrics.py, IMPROVEMENTS.md:120-130, AUDIT_REPORT.md:105-115
Error Tracking and Sentry
The system integrates with Sentry to provide real-time error monitoring. This integration captures unhandled exceptions and performance bottlenecks that bypass standard logging.
Error Handling Policy
In accordance with engineering mandates, the project avoids broad try/catch blocks for general flow control, allowing errors to propagate to the Sentry middleware. This ensures that every unexpected failure mode is captured with its full execution context, including local variables and the state of the AssetRelationshipGraph at the time of failure.
Sources: src/observability/sentry.py, GEMINI.md:8-12, AI_RULES.md:40-45
Security Observability
Observability extends to the security domain through automated scanning and audit trails. The CI/CD pipeline and runtime environment monitor for vulnerabilities and unauthorized state changes.
Security Tooling
- CodeQL: Performs deep static analysis to find security vulnerabilities in the source code.
- Bandit: Specifically scans Python code for common security issues.
- Safety: Checks installed dependencies against a database of known vulnerabilities.
- Dependency Review: Monitors new dependencies introduced in Pull Requests for security flaws.
Sources: AUDIT_REPORT.md:80-95, IMPROVEMENTS.md:100-110
Health and Readiness Checks
The application provides specialized endpoints to facilitate automated recovery and deployment gating.
- Liveness Probe: Indicates if the FastAPI application is running.
- Readiness Probe: Checks if the system is ready to serve traffic, verifying database connectivity and the presence of the
AssetRelationshipGraphin memory. - Hosted Readiness: A manual GitHub Actions workflow (
hosted-readiness.yml) performs bounded probes against deployments to verify connectivity before promotion.
Sources: ARCHITECTURE.md:200-215, IMPROVEMENTS.md:125-130, AUDIT_REPORT.md:150-160
Tracing and Observability ensure that the Financial Asset Relationship Database remains enterprise-ready. By combining request-level tracing, structured logging, and robust security monitoring, the system provides the necessary data for disaster recovery, performance tuning, and incident response.