Observability and Tracing
Relevant source files
The following files were used as context for generating this wiki page:
- packages/core/src/utils/observability/AgentObserver.ts
- packages/core/src/utils/observability/agentEvents.ts
- apps/web/src/features/trace-visualization/db/Workflow/retrieveWorkflow.ts
- apps/web/src/app/(protected)/(traces)/trace/[wf_inv_id]/page.tsx
- apps/web/src/app/(protected)/(traces)/trace/[wf_inv_id]/components/PerformanceOverview.tsx
- apps/web/src/app/(protected)/dev/components/PipelineTester.tsx
- packages/core/src/utils/persistence/README.md
The Observability and Tracing system provides a comprehensive framework for monitoring, debugging, and analyzing the behavior of agent-based workflows. It captures detailed events during a workflow's lifecycle, persists this data, and enables both real-time and post-hoc visualization. This allows developers to understand agent collaboration, identify performance bottlenecks, and track the evolutionary optimization of workflows.
The system is composed of two main parts: a backend event emission mechanism using an observer pattern, and a frontend application for retrieving and visualizing the resulting trace data. This includes detailed timelines, performance metrics, and live event streaming for development and testing.
Core Observability Architecture
The backend observability is built around an AgentObserver class that acts as an event emitter. It allows different parts of the system to subscribe to and publish structured AgentEvent objects, which capture significant occurrences during a workflow's execution.
AgentObserver
The AgentObserver is a singleton class that manages event listeners and dispatches events. It uses the standard EventEmitter from Node.js to provide a decoupled way for components to report their status and actions.
This diagram shows the structure of the AgentObserver class.
Agent Events
Events are strongly-typed objects conforming to the AgentEvent interface. Each event has a type and a payload specific to what it represents. This ensures consistency in the data being tracked.
| Event Type | Payload Description |
|---|---|
agent.start | Contains nodeId and invocationId when an agent node begins execution. |
agent.end | Contains nodeId, invocationId, and the final result of the agent's execution. |
agent.error | Contains nodeId, invocationId, and the error object if an agent fails. |
agent.tool.start | Contains nodeId, invocationId, and the toolCall details before a tool is executed. |
agent.tool.end | Contains nodeId, invocationId, toolCall, and the toolResult after a tool finishes. |
connected | A simple event to confirm a live stream connection is established. |
Sources: packages/core/src/utils/observability/agentEvents.ts:1-41, apps/web/src/app/(protected)/dev/components/PipelineTester.tsx:205-212
Trace Data Persistence and Retrieval
Trace data, including workflow and node invocations, is persisted to a Supabase database. The frontend application queries this data to reconstruct the execution history for visualization.
Database Schema for Tracing
The following diagram illustrates the key tables and their relationships involved in storing trace data. A WorkflowInvocation represents a single run of a specific WorkflowVersion. Each invocation consists of multiple NodeInvocation steps, which in turn can exchange Message objects.
Data Retrieval
The frontend retrieves all data for a specific trace using the retrieveWorkflowInvocation function. This function executes a series of Supabase queries to fetch the main invocation record, its version, all associated node invocations, and all messages exchanged during the run.
The data is fetched in a structured manner:
- Fetch the
WorkflowInvocationby its ID. - Fetch the corresponding
WorkflowVersion. - Fetch all
NodeInvocationrecords linked to theWorkflowInvocation. - Fetch all
Messagerecords linked to theWorkflowInvocation. - The results are then grouped and returned as a single
FullWorkflowResultobject.
This flow is visualized below.
Trace Visualization
The lucky web application provides a detailed trace visualization page that consumes the data retrieved from the database. This page displays performance metrics and a chronological timeline of the workflow execution.
Performance Overview
The PerformanceOverview component displays key metrics about the workflow run, providing a high-level summary of its execution.
| Metric | Description |
|---|---|
| Status | The final status of the workflow (completed, failed, running). |
| Total Cost | The total USD cost of the invocation, aggregated from all node invocations. |
| Total Duration | The total time taken from the start to the end of the workflow. |
| Total Nodes | The number of agent nodes executed in the timeline. |
| Bottleneck Node | The node with the longest execution duration. |
| Fitness Score | The score assigned by the evolutionary algorithm, if applicable. |
| Accuracy | The accuracy percentage from the fitness evaluation, if applicable. |
These metrics are calculated on the client-side based on the fetched trace data.
Sources: apps/web/src/app/(protected)/(traces)/trace/[wf_inv_id]/page.tsx:81-120, apps/web/src/app/(protected)/(traces)/trace/[wf_inv_id]/components/PerformanceOverview.tsx:1-300
Execution Timeline
The Timeline component renders the sequence of NodeInvocation events, showing the inputs, outputs, and metadata for each step in the workflow. This allows developers to trace the flow of data and logic through the agent graph. The data for the timeline is prepared by mapping the nodeInvocations array to FullTraceEntry objects.
Sources: apps/web/src/app/(protected)/(traces)/trace/[wf_inv_id]/page.tsx:30-79
Live Event Streaming
For development and testing, the PipelineTester component provides a UI to invoke a pipeline and view events in real-time using Server-Sent Events (SSE).
Streaming Flow
The sequence diagram below shows how the client initiates a pipeline run and subscribes to the live event stream.
- The client generates a random ID.
- It establishes an SSE connection to
/api/agents/{randomId}/stream. - It then sends a POST request to
/api/dev/pipeline/invokewith the configuration and the same random ID. - The backend pipeline uses the
AgentObserverto emit events. - A listener on the backend forwards these events to the corresponding SSE stream.
- The client receives and displays the events as they happen.
This mechanism provides immediate feedback during development, allowing developers to observe the agent's behavior, tool usage, and decision-making process in real time without waiting for the full trace to be persisted and retrieved.
Summary
The observability and tracing system is a critical component of the lucky framework. It provides deep insights into the execution of complex, multi-agent workflows through structured event logging, persistent storage, and powerful visualization tools. By offering both post-hoc analysis of full traces and real-time event streaming, it equips developers with the necessary tools to debug, optimize, and understand the emergent behaviors of their autonomous systems.