Observability & Tracing
Relevant source files
The following files were used as context for generating this wiki page:
Observability and Tracing in Flow provide a high-fidelity view into the execution of LangGraph-based agentic pipelines. The system tracks every execution, tool call, and retrieved document, mapping them into a structured knowledge graph to enable deep analysis of agent behavior and performance. This infrastructure is critical for the "Continuous Improvement Loop," where agents are automatically evaluated and refined based on historical execution data.
The observability suite includes structured logging, real-time streaming via Server-Sent Events (SSE), integration with LangSmith for external tracing, and a visual Knowledge Graph for inspecting relationships between executions, skills, and tools.
Sources: README.md:144-150, apps/web/src/components/marketing/HomeLanding.tsx:11-14
Execution Tracing & Logging
Flow utilizes a specialized callback handler, FlowCallbackHandler, to instrument LangGraph executions. This handler attaches contextual metadata to every event, ensuring that logs are traceable across different workspaces and agents.
Key Instrumentation Features
- Contextual Metadata: Every log event includes the
workspace_id,agent_id,execution_id, and the specifictemplateused. - Latency Tracking: Per-call latency is measured in milliseconds for both LLM generations and tool invocations.
- Structured Events: The system emits structured
structlogevents for lifecycle stages:llm.start,llm.end,tool.start,tool.end, andchain.error. - External Integration: Traces can be exported to LangSmith using the
FLOW_LANGSMITH_API_KEY.
Sources: README.md:144-152
Logging Configuration
The system supports JSON-formatted logs for containerized environments, enabled via the FLOW_LOG_JSON=true environment variable. This allows for structured querying of failures, particularly for background cron jobs which log exc_type, workspace_id, and agent_id.
Sources: README.md:153-155, services/api/scripts/seed_skills.py:650-652
Visual Knowledge Graph
The platform features an "Agentic Knowledge Graph" that visualizes the relationships between different system entities. This graph serves as a debugging and discovery tool for developers to see how agents utilize skills and tools during executions.
The diagram shows the hierarchical and functional relationships between core entities tracked in the observability graph.
Sources: apps/web/src/app/(app)/graph/page.tsx:35-42, apps/web/src/components/marketing/HomeLanding.tsx:11-14
Graph Node Types & Metadata
The graph categorizes information into several distinct node types, each with specific metadata and visual coding:
| Node Type | Color Coding | Purpose | Metadata Captured |
|---|---|---|---|
| Execution | #bbf7d0 | A single run of an agent | Status, start time, result |
| Agent | #818cf8 | The configured AI entity | Template, status, ref_id |
| Skill | #fbbf24 | Anthropic-style instruction blocks | Version, score, use_count, triggers |
| Tool Call | #f472b6 | Specific invocation of a tool | Tool name, arguments, latency |
| Genome Version | #94a3b8 | A snapshot of agent config | Model, provider, status |
Sources: apps/web/src/app/(app)/graph/page.tsx:44-55, apps/web/src/app/(app)/graph/page.tsx:430-495
Real-time Observability via SSE
Flow uses Server-Sent Events (SSE) to stream execution progress to the frontend. This provides immediate visibility into the LangGraph pipeline's internal state transitions.
Sequence of events during a streamed execution, showing how internal pipeline transitions are surfaced to the user.
Sources: README.md:79-84, AGENTS.md:30-32
Streaming Features
- Zero-latency Streaming: Built on React 19 to handle raw SSE streams with high performance.
- Pipeline Transparency: The UI displays the active node (e.g.,
planner → worker → synthesizer). - Citation Markers: Streaming events include inline citation markers
[1], which link directly to the source chunks retrieved by the worker.
Sources: README.md:83-85, apps/web/src/components/marketing/HomeLanding.tsx:19-21
Evaluation & Metrics
Observability extends into the quality dimension through "Golden Sets" and automated evaluation.
Performance Metrics
The system tracks and aggregates scores from "LLM Judges" to provide performance overviews for specific agent versions:
- Avg Score: The mean score (0.0–1.0) across all items in a Golden Set.
- Pass Rate: The percentage of items achieving a score ≥ 0.7.
- Regression Tracking: Historical eval runs are grouped by
eval_run_idto track quality changes over time.
Sources: services/api/flow/interfaces/http/routes/golden_sets.py:315-325, services/api/flow/interfaces/http/routes/golden_sets.py:350-360
Diagnostic Feedback Loop
Scores below 50% trigger the creation of an "Agent Negative." This captures the specific query that resulted in a poor outcome. These negatives are used as few-shot examples for the retrieval grader in future runs to prevent repeat failures.
Sources: README.md:162-165
Conclusion
The observability and tracing system in Flow is not merely for debugging but acts as the foundation for the project's self-improving nature. By combining structured technical logs with a visual knowledge graph and automated evaluation metrics, Flow provides developers with the necessary transparency to move agentic workflows from experimental scripts to production-ready systems.
Sources: README.md:144-150, apps/web/src/components/marketing/HomeLanding.tsx:11-21