LLM & GenAI Instrumentation
Relevant source files
The following files were used as context for generating this wiki page:
Pydantic Logfire provides specialized instrumentation for Large Language Models (LLMs) and Generative AI frameworks to offer deep observability into AI-driven applications. This system captures critical data such as prompts, completions, token usage, and model parameters, allowing developers to monitor and evaluate the performance and cost of their GenAI integrations.
The instrumentation is designed to be non-intrusive, leveraging OpenTelemetry-based tracing to wrap popular SDKs like OpenAI, Anthropic, and Google GenAI. It integrates seamlessly with the broader Logfire ecosystem, including the Web UI's LLM Panels for specialized visualization of chat sequences and model outputs.
Sources: mkdocs.yml:104-124, logfire-api/logfire_api/init.py:183-195
Supported Providers and Frameworks
Logfire supports a wide range of LLM providers and GenAI frameworks through dedicated instrumentation functions. These functions wrap the underlying clients to automatically generate spans for every API call.
| Provider / Framework | Instrumentation Function | Dependencies |
|---|---|---|
| OpenAI | instrument_openai() | openai |
| Anthropic | instrument_anthropic() | anthropic |
| Google GenAI | instrument_google_genai() | google-genai |
| LiteLLM | instrument_litellm() | litellm |
| Pydantic AI | instrument_pydantic_ai() | pydantic-ai-slim |
| DSPy | instrument_dspy() | dspy |
| OpenAI Agents | instrument_openai_agents() | openai-agents |
Sources: pyproject.toml:79-82, logfire-api/logfire_api/init.py:183-195
Instrumentation Architecture
The instrumentation logic typically follows a standard pattern where the Logfire SDK wraps the provider's client or global module methods. This allows Logfire to intercept requests and responses to extract metadata.
Data Flow for LLM Requests
When an instrumented LLM call is made, Logfire performs the following steps:
- Starts a new OpenTelemetry span labeled with the provider and method (e.g.,
openai.chat.completions). - Captures request attributes such as
model,temperature, and the input messages/prompts. - Executes the actual LLM API call.
- On response, extracts token usage (prompt, completion, and total tokens) and the generated content.
- Records any exceptions as span events.
The diagram above illustrates how Logfire wraps the standard LLM provider request cycle to extract telemetry without changing the application's core logic.
Sources: logfire/_internal/cli/run.py:102-125, logfire-api/logfire_api/init.py:42-100
CLI-Based Auto-Instrumentation
The Logfire CLI provides a run command that can automatically detect and instrument installed LLM packages. This allows developers to add observability to existing scripts without modifying the source code.
Detection Logic
The CLI maps installed packages to their corresponding Logfire instrumentation methods. For example, if the openai package is detected in the environment, the CLI will call logfire.instrument_openai().
OTEL_INSTRUMENTATION_MAP = {
# ... other integrations
'pydantic-ai-slim': 'pydantic_ai',
'anthropic': 'anthropic',
'openai': 'openai',
'openai-agents': 'openai_agents',
}
Sources: logfire/_internal/cli/run.py:53-57
Implementation in logfire run
The instrument_packages function iterates through detected packages and dynamically invokes the instrumentation.
def instrument_packages(installed_otel_packages: set[str], instrument_pkg_map: dict[str, str]) -> list[str]:
instrumented: list[str] = []
# ...
for otel_pkg_name in installed_otel_packages:
import_name = instrument_pkg_map[otel_pkg_name]
try:
instrument_package(import_name)
except Exception:
continue
instrumented.append(base_pkg)
return instrumented
Sources: logfire/_internal/cli/run.py:102-125
Integration with Evaluation Systems
LLM instrumentation is closely linked to Logfire's evaluation (Evals) and Dataset features. By capturing the inputs and outputs of LLM spans, Logfire enables the creation of datasets for backtesting and performance tracking.
- Datasets: Captured LLM traces can be exported to datasets for further analysis.
- Evals: Instrumented spans provide the raw data required for automated or manual evaluation of model responses against ground truth.
Sources: mkdocs.yml:146-160, pyproject.toml:83
Configuration and Options
While many LLM instrumentations work out-of-the-box, they are influenced by global Logfire configuration settings defined in LogfireConfig.
Key Configuration Parameters
scrubbing: Sensitive data in prompts or completions can be redacted usingScrubbingOptions.sampling: High-volume LLM applications can useSamplingOptionsto control the number of traces sent to the backend.inspect_arguments: Enables enhanced inspection of f-string arguments in log calls, which can be useful when logging LLM prompts.
Sources: logfire/_internal/config.py:315-380, logfire/_internal/config.py:537-545
LLM Dashboard Panels
The data captured via instrumentation is specifically formatted to populate "LLM Panels" in the Logfire Web UI. These panels provide a specialized view of chat histories, including system prompts, user messages, and assistant responses, along with token cost summaries.
Sources: mkdocs.yml:138