Experimental Features: Datasets SDK
Relevant source files
The following files were used as context for generating this wiki page:
The Datasets SDK is an experimental module within Pydantic Logfire designed to manage datasets for evaluation purposes, particularly in AI and LLM workflows. It allows developers to programmatically create, update, and manage collections of test cases, which can include inputs, expected outputs, and metadata for evaluation.
This module is currently housed under the experimental namespace, indicating that while functional, the API and features are subject to change. It relies on the pydantic-evals package for certain core data structures like Case and Dataset.
Sources: pyproject.toml:98, tests/test_datasets_client.py:12-19, logfire/experimental/datasets/init.py
Architecture and Core Clients
The Datasets SDK provides both synchronous and asynchronous clients for interacting with the Logfire Datasets API. These clients handle authentication via API keys and manage the serialization of Python objects (dataclasses and Pydantic models) into JSON schemas for storage.
Client Comparison
| Feature | LogfireAPIClient | AsyncLogfireAPIClient |
|---|---|---|
| Execution | Synchronous | Asynchronous (anyio/asyncio) |
| HTTP Backend | httpx.Client | httpx.AsyncClient |
| Authentication | API Key / Bearer Token | API Key / Bearer Token |
| Serialization | Automatic (Pydantic/Dataclasses) | Automatic (Pydantic/Dataclasses) |
Sources: tests/test_datasets_client.py:91-110, tests/test_datasets_client.py:270-323
Communication Flow
The following diagram illustrates how the client interacts with the Datasets API to manage resources.
The SDK converts Python types to JSON schemas to ensure the API understands the structure of inputs and outputs.
Sources: tests/test_datasets_client.py:113-140
Data Models and Serialization
The SDK utilizes several key structures to represent data. The primary unit of data is the Case, which represents a single entry in a dataset.
Case Structure
A Case typically consists of:
- Name: An optional unique identifier for the case.
- Inputs: The data to be processed (e.g., an LLM prompt).
- Expected Output: The gold-standard result for evaluation.
- Metadata: Additional context (e.g., source of the data).
- Evaluators: References to specific evaluation logic to be applied.
Sources: tests/test_datasets_client.py:65-71, tests/test_datasets_client.py:196-224
Schema Conversion
The SDK includes internal utilities to convert Python type hints into JSON schemas. This allows the backend to enforce data consistency in the UI and during evaluations.
# Internal utility for type-to-schema conversion
def _type_to_schema(typ):
# Converts Dataclasses or Pydantic models to JSON Schema
...
# Example of manual dataset creation with types
client.create_dataset(
name='eval-set',
input_type=MyInputClass,
output_type=MyOutputClass
)
Sources: tests/test_datasets_client.py:113-131, tests/test_datasets_client.py:330-345
Key API Endpoints and Methods
The clients expose methods that map directly to the REST API endpoints of the Datasets service.
| Method | Endpoint | Purpose |
|---|---|---|
list_datasets() | GET /v1/datasets/ | Retrieves all datasets accessible by the token. |
create_dataset() | POST /v1/datasets/ | Initializes a new dataset with optional schemas. |
add_cases() | POST /v1/datasets/{name}/cases/bulk/ | Uploads multiple cases in a single request. |
export_dataset() | GET /v1/datasets/{name}/export/ | Downloads the entire dataset for local use. |
delete_case() | DELETE /v1/datasets/{name}/cases/{id}/ | Removes a specific case from a dataset. |
Sources: tests/test_datasets_client.py:75-90, tests/test_datasets_client.py:325-360
Future Roadmap
Several enhancements are planned for the Datasets SDK to improve developer experience and backend reliability:
High-Level Typed API
Future versions aim to provide a DatasetHandle or AsyncDatasetHandle. This would allow developers to interact with a specific dataset object rather than passing the dataset name to every client call.
Sources: plans/datasets-future-work.md:5-12
Idempotency and Upserts
To prevent duplicate cases, an add_cases idempotency feature is planned. This involves a backend unique constraint on (dataset_id, name) and an on_conflict parameter in the SDK to handle existing cases (e.g., update, skip, or error).
Sources: plans/datasets-future-work.md:14-19
Known Issues and Improvements
- Schema Validation: Moving validation server-side to prevent the UI from saving cases that violate the dataset's JSON schema.
- Metadata in Lists: Enhancing the
list-casesresponse to include metadata, avoiding the need for individualget-casecalls. - Export/Import Parity: Unifying the structures used for exporting data and re-importing it via the API.
Sources: plans/datasets-future-work.md:25-35
Usage Example
The following example demonstrates how to use the synchronous client to create a dataset and add cases.
from logfire.experimental.datasets import LogfireAPIClient
from pydantic_evals import Case
client = LogfireAPIClient(api_key='your_key')
# Add cases with tags
cases = [
Case(inputs={'question': '2+2'}, expected_output={'answer': '4'}),
Case(inputs={'question': '3+3'}, expected_output={'answer': '6'})
]
client.add_cases('math-dataset', cases, tags=['sprint-1'])
Sources: tests/test_datasets_client.py:440-455, logfire/experimental/datasets/init.py
The Datasets SDK bridges the gap between raw telemetry and structured evaluation, enabling a "Know more. Build faster" approach for AI application development.
Sources: README.md:1