Observability & Monitoring
Relevant source files
The following files were used as context for generating this wiki page:
Observability and monitoring in Infrahub provide visibility into system health, user activity, and background task execution. The system collects anonymous usage telemetry, maintains audit trails for all data mutations, and surfaces diagnostic information through pipeline task logs.
These capabilities allow administrators to track database scale, identify performance bottlenecks via benchmark tracking, and debug automated workflows like artifact generation.
Anonymous Usage Telemetry
Infrahub collects anonymous usage telemetry to track platform health and scale. The collection mechanism focuses on high-level usage metrics rather than sensitive data. Telemetry is windowed and subject to retention and degradation policies.
Telemetry Payload Contract
The telemetry collection service transmits a specific payload contract that captures the state of the instance:
| Field Category | Metric Name | Description |
|---|---|---|
| Accounts | accounts.active | The number of active user accounts |
| Activity | activity_24h.* | Various activity metrics captured over a 24-hour window |
| Database | database.node_count.corenode | Total count of core nodes within the graph database |
Sources: specs/telemetry-collection-infp-589/checklists/requirements.md:32-34, backend/AGENTS.md:139-141
Telemetry Flow
The following diagram illustrates the flow of telemetry data from the backend to the collection endpoint:
The Telemetry Collector aggregates database node counts and account status, windows the activity data, and transmits a technology-agnostic payload.
Sources: specs/telemetry-collection-infp-589/checklists/requirements.md:31-36
Diagnostic & Pipeline Visibility
Infrahub surfaces diagnostic information during automated processes, particularly within the proposed change pipeline. This visibility helps users understand why specific actions, such as artifact regeneration, occur.
Trigger Identification
When the system regenerates artifacts, it logs the specific cause in the pipeline task log. This identifies the "why" trail for developers and SREs. Triggers include:
- Query Changes: Modifications to the GraphQL query associated with an artifact.
- File Modifications: Changes to transform source files or transitive dependencies.
- Attribute Updates: Changes to the artifact definition node itself (e.g., target group repointing).
- Unresolved References: Diagnostic logs identify non-literal Jinja2 includes that require a
watch:declaration.
Sources: specs/infp-409-artifact-regen-triggers/spec.md:95-99, specs/infp-409-artifact-regen-triggers/spec.md:151-155
Diagnostic Sequence
The system follows a specific sequence to log regeneration decisions:
The selection gate identifies the specific modification and logs the trigger before initiating worker tasks.
Sources: specs/infp-409-artifact-regen-triggers/spec.md:200-204
Audit Trails & Metadata
The system enforces data integrity and traceability through mandatory audit metadata on all nodes and relationships.
- Temporal Metadata: Every mutation tracks
created_atandupdated_attimestamps. - Identity Tracking: The system records
created_byandupdated_byfor all mutations on global and default branches. - Branch-Specific Audit: Metadata is maintained across Infrahub's branching model to ensure version-control guarantees.
Sources: dev/constitution.md:129-131, dev/constitution.md:43-46
Performance & Health Monitoring
Infrahub uses several mechanisms to monitor and protect system performance, ranging from CI-level benchmarks to runtime admission control.
Benchmark Tracking
Performance regressions are detected during the development lifecycle:
- pytest-benchmark: Used for performance-sensitive code paths.
- CodSpeed CI: Monitors performance regressions in continuous integration.
- Query Benchmarks: Dedicated tests in
tests/query_benchmark/validate Cypher efficiency.
Sources: dev/constitution.md:117-118, dev/constitution.md:144-146
API Health & Backpressure
The backend includes a priority-aware load-shedding layer to maintain availability under stress:
- Admission Middleware: Implements API backpressure using CoDel and slot pools.
- Database-Stress Signal: Monitoring signals from the database trigger load shedding to prevent system saturation.
- Priority Lanes: Prefect workflows use priority lanes to manage asynchronous task health.
Sources: backend/AGENTS.md:130-132, backend/AGENTS.md:120-122, backend/AGENTS.md:134-136
Monitoring Dashboards
Infrahub provides pre-configured dashboards for real-time monitoring:
- Grafana Dashboards: Located in
development/grafana/provisioning/dashboards/. - Infrahub Monitoring: A centralized dashboard provides a high-level overview of instance health.
Sources: AGENTS.md:183-185, backend/AGENTS.md:16-18
Summary
Observability in Infrahub is structured across telemetry for platform scale, diagnostic logging for pipeline transparency, and audit metadata for data integrity. The combination of Grafana dashboards and diagnostic task logs ensures that both system-wide health and granular task execution are visible to operators and developers. Standard performance benchmarks and admission control mechanisms further protect the system from degradation.