OpenDPI Implementation

Relevant source files

The following files were used as context for generating this wiki page:

• internal/opendpi/parse.go
• internal/opendpi/writer.go
• internal/opendpi/parse_test.go
• internal/commands/ports_describe.go
• internal/session/testdata/valid/spec/opendpi.yaml

The OpenDPI implementation within the Daco CLI provides a robust framework for parsing, validating, and writing data product specifications. It acts as the core engine for interpreting how data "Ports" (logical data interfaces) connect to physical "Connections" (infrastructure like PostgreSQL or Kafka) and how their data structures are defined via JSON Schema.

This module handles the full lifecycle of a specification, from reading YAML/JSON files and resolving complex $ref architectures (including local and external file references) to serializing these structures back to disk. It serves as the primary data model used by other CLI components, such as translators and descriptive commands.

Sources: internal/opendpi/parse.go:1-20, internal/opendpi/writer.go:1-15

Data Model Architecture

The OpenDPI specification is structured into several key entities that define the metadata, connectivity, and structural contracts of a data product.

Core Entities

Entity	Description
Spec	The root object containing the OpenDPI version, info, tags, connections, and ports.
Info	Metadata about the data product including title, version, and description.
Connection	Defines infrastructure access, including type (e.g., postgresql), host, and variables.
Port	A logical data interface that maps a schema to one or more physical connections.
Schema	A JSON Schema definition (inline or referenced) describing the data format.

Sources: internal/opendpi/parse.go:211-255, internal/session/testdata/valid/spec/opendpi.yaml:1-25

Entity Relationships

The following diagram illustrates how Ports act as the bridge between logical schemas and physical infrastructure connections.

The Port entity uses a PortConnection structure to link specific locations (like table names) to defined connections.
Sources: internal/opendpi/parse.go:240-250, internal/opendpi/writer.go:45-65

Parsing and Reference Resolution

The Parser is responsible for transforming raw YAML or JSON into a fully resolved Spec object. This process includes heavy lifting for JSON Schema references.

Resolution Logic

1. Raw Parsing: Loads the initial file into a rawSpec structure.
2. External Loading: If a Port or Component schema uses a file reference (e.g., $ref: "schemas/user.yaml"), the parser uses a jschema.Loader to fetch the file from the filesystem.
3. Component Mapping: Resolves internal references starting with #/components/schemas/.
4. Reference Rewriting: Converts various reference formats into a unified #/$defs/ structure for internal consistency.
5. Unified Schema Map: Aggregates all resolved schemas into a single map for easy lookup by the CLI.

Sources: internal/opendpi/parse.go:34-150, internal/opendpi/parse_test.go:85-120

Parser Flow

Serialization and Output

The Writer component handles the conversion of the internal Spec back into a persistent file format. It supports both JSON and YAML output.

Key Functions

• Writer.Write(spec, specDir): Serializes a Spec to a file named opendpi.<ext> in the target directory.
• toRaw(spec): A conversion function that prepares the internal memory model for serialization, including reconstructing the reverse lookup for connection references.
• WriteSchemaFile: Generates a skeleton JSON Schema (type: object) for new ports.

Sources: internal/opendpi/writer.go:26-75, internal/opendpi/writer.go:107-115

Writing Process Sequence

Inspection and Interaction

The CLI provides tools to interact with the OpenDPI model, specifically through the ports describe command. This command utilizes the Spec model to provide human-readable or machine-readable (JSON/YAML) views of the implementation.

Display Logic

The command iterates through the Port properties and its associated Schema. It recursively prints the schema tree, handling:

• Primitive types (string, integer, etc.)
• Formats (date-time, uuid)
• Required fields
• Nested properties and array items
• Definitions ($defs)

Sources: internal/commands/ports_describe.go:124-210

Implementation Examples

Minimal Specification

A basic OpenDPI document requires at least the version, info, one connection, and one port.

opendpi: "1.0.0"
info:
  title: My Data Product
  version: "1.0.0"
connections:
  database:
    type: postgresql
    host: localhost:5432
ports:
  users:
    connections:
      - connection: "#/connections/database"
        location: users
    schema:
      type: object
      properties:
        id: {type: integer}

Sources: internal/session/testdata/valid/spec/opendpi.yaml:1-25

Handling Circular References

The parser is designed to handle self-referential schemas (e.g., a "Node" that contains a "parent" of type "Node") without hanging or panicking. It ensures the schema is added to definitions once and correctly linked.
Sources: internal/opendpi/parse_test.go:275-300

Summary

The OpenDPI implementation provides a structured way to manage the contract between data producers and consumers. By decoupling logical data ports from physical connection details and utilizing industry-standard JSON Schema for structural definitions, it enables the Daco CLI to perform complex tasks like schema translation and infrastructure mapping with high technical accuracy.