Data & Observability

Relevant source files

The following files were used as context for generating this wiki page:

• architecture/22.3-data-observability-tooling-catalog.md
• architecture/22.6-standard-library-implementation-plan.md
• architecture/decisions/0031-public-library-tiers-and-names.md
• architecture/decisions/0032-concise-apis-and-cost-contracts.md
• crates/tools/architecture-tests/src/tests.rs
• architecture/14-libraries-namespaces-and-loading.md

The Data & Observability system in Pop Lang encompasses a suite of namespaces and official extension packages designed for structured data handling, persistence, and system monitoring. This module is categorized under the "Data, Observability, and Tooling" catalog and prioritizes concise APIs with explicit cost contracts to ensure high performance and static safety.

The primary goal of this system is to provide native, strongly-typed abstractions for common data formats (JSON, YAML, TOML), database interactions (SQL), and telemetry (logs, traces, metrics) without the overhead of dynamic lookup or runtime reflection.

Sources: architecture/22.3-data-observability-tooling-catalog.md:1-5, architecture/decisions/0032-concise-apis-and-cost-contracts.md:7-12

Data Management

Data management in Pop Lang is divided between the core Pop.Standard library and the official Pop.Data extension package. The system uses a "data-first" approach, favoring records and tagged unions for structured data representation.

Structured Formats and Codecs

Pop Lang provides direct root namespaces for common data formats, ensuring that qualified names remain brief and discoverable. Shared logic for schema handling and binary encoding resides in the Codec namespace.

Namespace	Purpose	Tier
Json	JSON encoding and decoding	Standard
Yaml	YAML 1.2 processing	Standard
Toml	TOML configuration processing	Standard
Xml	XML 1.0 fragment and document processing	Standard
Csv	RFC 4180 CSV processing	Standard
Codec	Shared schema, binary-codec, and streaming primitives	Standard

Sources: architecture/decisions/0031-public-library-tiers-and-names.md:11-18, architecture/22.3-data-observability-tooling-catalog.md:12-25

Persistence and SQL

Persistence is primarily handled through the Pop.Data package, which provides typed adapters for SQL databases and key-value stores.

• Data: The root of the official extension package for data access.
• Sql: Provides typed SQL query execution, transaction management, and driver-neutral protocols.
• Store: Interface for structured key-value and document storage.

The following diagram illustrates the relationship between the Pop.Data extension and its sub-namespaces:

The diagram shows the internal structure of the Pop.Data official extension package.

Sources: architecture/decisions/0031-public-library-tiers-and-names.md:20-22, architecture/22.6-standard-library-implementation-plan.md:26-27, crates/tools/architecture-tests/src/tests.rs:163-172

Observability and Telemetry

Observability is centered around the Telemetry namespace, which consolidates logs, traces, and metrics into a single, cohesive API root.

Telemetry Architecture

The Telemetry system is designed to be efficient, with primitives defined in early implementation phases to support integration across the standard library.

• Logs: Structured logging with severity and category support.
• Traces: Distributed tracing capabilities.
• Metrics: Counters, gauges, and histograms for system monitoring.

The implementation plan requires telemetry primitives to be established before broader integrations, ensuring that redaction and deterministic capture are handled consistently.

Sources: architecture/22.3-data-observability-tooling-catalog.md:27-31, architecture/22.6-standard-library-implementation-plan.md:26-27

Diagnostics and Logging Flow

Diagnostics in Pop Lang are treated as structured semantic APIs. The Diagnostic namespace provides stable codes (POP####) and versioned machine schemas, ensuring that human-readable text and machine-readable data (like JSON or SARIF) are derived from the same objects.

This flow illustrates how internal system issues are transformed into structured diagnostic objects for both human and machine consumption.

Sources: architecture/decisions/0032-concise-apis-and-cost-contracts.md:37-39, architecture/22.3-data-observability-tooling-catalog.md:41-45

Cost Contracts and Performance

A defining characteristic of Data & Observability in Pop Lang is the explicit documentation of performance costs. ADR 0032 mandates that every public operation must document its "cost class."

Cost Categories

The following table summarizes the metrics reported for data and observability operations:

Metric	Requirement
Allocation	Must state if the operation allocates or uses caller-owned buffers.
Copying	Distinguishes between owning values and non-owning views/slices.
Dispatch	Statically resolved or explicit interface/function-value calls.
Complexity	Documented asymptotic cost (e.g., O(n)).
Native Transitions	Explicitly identifies boundaries between the Pop runtime and native host.

Sources: architecture/decisions/0032-concise-apis-and-cost-contracts.md:14-22, architecture/decisions/0032-concise-apis-and-cost-contracts.md:49-60

Usability Rules

To maintain the Luau-shaped simplicity of the language, the API design rejects excessive ceremony like "Managers" or "Providers."

-- PREFERRED: Concise and direct
local user = Json.decode(text, UserSchema)
local results = Sql.query(db, "SELECT * FROM users")
Telemetry.log(.info, "User loaded", { id = user.id })

-- REJECTED: Ceremonial/OOP-shaped
local decoder = Data.Json.Decoder.new()
local user = decoder.decodeValue(text, UserSchema)

Sources: architecture/decisions/0032-concise-apis-and-cost-contracts.md:26-35

Implementation Status

The implementation of Data & Observability follows a multi-phase plan, prioritizing core formats before moving to specialized database and telemetry drivers.

1. Phase 2 (Formats): Json, Yaml, Xml, Csv, Toml, and Codec streaming primitives.
2. Phase 5 (Extensions): Pop.Data (SQL/Store) and Telemetry primitives.
3. Phase 6 (Tooling): Diagnostic and Documentation public schemas.

Sources: architecture/22.6-standard-library-implementation-plan.md:14-30

Conclusion

The Data & Observability module provides the bedrock for production-grade applications in Pop Lang. By strictly defining namespaces, enforcing explicit cost contracts, and utilizing official extension packages for heavyweight operations, the system maintains the language's core principles: high-level readability paired with low-level predictability and performance.