Port Management

Relevant source files

The following files were used as context for generating this wiki page:

• internal/commands/ports_add.go
• internal/commands/ports_list.go
• internal/commands/ports_describe.go
• internal/commands/ports_translate.go
• internal/opendpi/parse.go
• internal/opendpi/writer.go

Port Management is a core subsystem of the Daco CLI responsible for defining, listing, inspecting, and transforming data ports within an OpenDPI specification. In the context of a Data Product, a "Port" represents a specific data interface—such as a database table or a Kafka topic—coupled with a structural schema that defines the data format.

The system allows users to manage the lifecycle of these ports through interactive CLI commands or non-interactive flags. This includes creating new port definitions, associating them with underlying infrastructure connections (e.g., PostgreSQL, Kafka), and translating their schemas into various target formats like PySpark, Protobuf, or Go types.

Sources: internal/commands/ports_add.go, internal/commands/ports_describe.go, internal/commands/ports_translate.go

Core Data Structures

Port management relies on the Port and Spec structures defined within the opendpi package. A port links a human-readable description and a JSON Schema to one or more physical locations via a connection.

The Spec acts as the root container, holding all defined ports and their associated connections. Each Port contains a Schema (referenced via $ref or defined inline) and a list of PortConnection objects that map the logical port to a physical Location (e.g., a specific table name or topic).

Sources: internal/opendpi/parse.go:193-248, internal/opendpi/writer.go:41-76

Port Operations

Adding Ports

Adding a port involves updating the OpenDPI specification file and generating a corresponding schema file. The CLI supports both an interactive form-based approach and a flag-driven non-interactive approach.

When a port is added, the system automatically creates a placeholder schema file in a schemas/ directory relative to the specification. This file is initialized as an empty JSON Schema object.

Sources: internal/commands/ports_add.go:66-135, internal/opendpi/writer.go:111-125

Listing and Describing Ports

The list command provides a summary of all ports, including their names, schema types, and associated connections. The describe command provides a deep dive into a specific port, rendering its full schema tree and connection details.

Feature	Description	Formats
List	Overview of all ports in the spec.	Table, JSON, YAML
Describe	Detailed view of a single port, including properties, required fields, and connection hosts.	Text, JSON, YAML

Sources: internal/commands/ports_list.go:48-95, internal/commands/ports_describe.go:102-150

Schema Translation

The translation system converts OpenDPI port schemas into code-ready formats. This is managed by the ports translate command, which utilizes a registry of available translators.

Schema Resolution Logic

The Port Management system employs a complex parser to resolve schema references ($ref). This allows ports to use shared components or external files.

1. Component Resolution: Schemas residing in #/components/schemas/ are resolved and mapped into the port's local $defs.
2. External File Loading: References to local files (e.g., schemas/user.yaml) are loaded from the filesystem and injected into the spec.
3. Circular Reference Handling: The parser is designed to handle self-referential schemas without entering infinite loops.
4. Ref Rewriting: Internal references are rewritten from OpenDPI component paths to standard JSON Schema $defs paths during processing.

Sources: internal/opendpi/parse.go:39-140

Command Summary

Command	Usage	Description
add	daco ports add [flags]	Creates a new port and a starter schema file.
list	daco ports list [-o format]	Displays all ports in the current specification.
describe	daco ports describe [name]	Shows full details of a port, including schema properties.
translate	daco ports translate -f [format]	Generates code (PySpark, Go, etc.) from the port schema.

Sources: internal/commands/ports_add.go:21-42, internal/commands/ports_list.go:23-42, internal/commands/ports_describe.go:28-60, internal/commands/ports_translate.go:31-52

The Port Management system ensures that data product interfaces are strictly defined and easily consumable by different technical stacks through its integrated validation and translation pipeline.