Internal SDK Testing & VCR Cassettes

Relevant source files

The following files were used as context for generating this wiki page:

• CLAUDE.md
• pyproject.toml
• tests/test_configure.py
• tests/test_logfire_api.py
• plans/datasets-future-work.md
• tests/otel_integrations/cassettes/test_openai/test_openrouter_streaming_reasoning.yaml

Introduction

The Logfire SDK employs a robust testing architecture designed to validate its opinionated wrapper around OpenTelemetry signals (traces, metrics, and logs). The testing suite ensures that instrumentation across various Python packages remains accurate and that the SDK correctly handles telemetry data before exporting it to the Logfire platform.

The system relies heavily on automated snapshots for asserting span structures and VCR-style recording for managing HTTP interactions with external APIs. This approach allows for deterministic testing of complex, asynchronous, and network-dependent features like AI model streaming and dataset management.

Sources: CLAUDE.md:1-8, CLAUDE.md:31-48

Core Testing Architecture

Logfire's testing strategy is built around specialized exporters and snapshotting tools that intercept OpenTelemetry data.

TestExporter and Inline Snapshots

The TestExporter class is the primary tool for validating span creation. It normalizes non-deterministic fields such as timestamps and IDs to ensure test stability. Developers use inline-snapshot to automatically generate and update expected span dictionaries.

The diagram shows how the TestExporter acts as a sink for telemetry data, processing it for comparison against stored snapshots.

Sources: CLAUDE.md:31-48, tests/test_configure.py:91-100

VCR Cassettes and HTTP Recording

For tests involving network calls (e.g., OpenAI integrations or Datasets API), Logfire uses pytest-recording to capture HTTP interactions in YAML "cassettes". These cassettes allow tests to run without active internet connections and provide reproducible responses.

Component	Purpose	Dependency
pytest-recording	Plugin used to record/play back HTTP sessions	pyproject.toml
VCR Cassettes	YAML files storing requests/responses	tests/otel_integrations/cassettes/
requests-mock	Used for simpler unit testing of config and auth flows	tests/test_configure.py

Sources: CLAUDE.md:52-56, pyproject.toml:139-140, tests/otel_integrations/cassettes/test_openai/test_openrouter_streaming_reasoning.yaml:1-10

Testing Infrastructure and Tooling

The SDK leverages several key libraries to maintain high code quality and technical accuracy across different Python environments.

Testing Dependencies

The project defines a comprehensive set of dev dependencies in pyproject.toml to support its multi-faceted testing approach.

• Assertion Tools: dirty-equals for flexible matching and inline-snapshot for data structure validation.
• Mocking: requests-mock for API simulation.
• Environment Validation: pyright for type checking and ruff for linting.
• Specialized Testing: pytest-asyncio for asynchronous tests and pytest-xdist for parallel execution.

Sources: pyproject.toml:98-155, CLAUDE.md:14-17

Logfire-API No-Op Validation

A critical part of the testing infrastructure is verifying the logfire-api shim. This package must remain a no-op when the full SDK is absent but correctly delegate to the main logfire package when it is present.

The sequence diagram illustrates the conditional behavior of the logfire-api package during runtime tests.

Sources: logfire-api/logfire_api/init.py, tests/test_logfire_api.py:38-55

Specialized Testing Scenarios

Configuration and Initialization Tests

Tests in tests/test_configure.py validate the complex initialization logic of the SDK, including environment variable overrides, pyproject.toml parsing, and project authentication.

• Service Metadata: Validates that service.name and service.version are correctly extracted from the environment or Git.
• Exporter Pipelines: Ensures that multiple tokens create distinct export pipelines for project migration scenarios.
• AWS Lambda: Specific tests for force_flush behavior in serverless environments to prevent span dropping.

Sources: tests/test_configure.py:348-375, tests/test_configure.py:840-880

Dataset and Evaluation Testing

Testing for experimental features like Datasets uses a mix of mock transports and plans to migrate toward full VCR cassette integration.

• Schema Validation: Tests ensure that Pydantic models and dataclasses are correctly converted to JSON schemas for the backend.
• Idempotency: Future work includes testing upsert behavior to prevent duplicate case creation.
• Mock Transport: Current implementation uses httpx.MockTransport to simulate the Datasets API.

Sources: plans/datasets-future-work.md:19-22, tests/test_datasets_client.py:94-118

Maintenance and Release Testing

Before a release, the SDK undergoes several automated steps to ensure the integrity of the public API.

1. Stub Generation: Running make generate-stubs to update .pyi files.
2. API Consistency: test_logfire_api.py checks that every member in logfire.__all__ has a corresponding entry in the logfire-api shim.
3. Documentation Examples: make test-update-examples validates that code snippets in the docs are functional and formatted correctly.

Sources: CLAUDE.md:58-64, CONTRIBUTING.md:12-14, tests/test_logfire_api.py:58-150

Conclusion

The Logfire internal testing system effectively combines traditional unit testing with advanced data validation techniques. By utilizing TestExporter for span structure, VCR cassettes for API interaction, and strict no-op validation for the logfire-api shim, the project maintains high reliability across its diverse set of integrations and execution environments.