Artifact Registration

Relevant source files

The following files were used as context for generating this wiki page:

• internal/commands/ports_describe.go
• internal/opendpi/parse.go
• internal/opendpi/writer.go
• internal/opendpi/parse_test.go
• CONTRIBUTING.md
• cmd/gendocs/main.go

Artifact Registration in the Daco CLI refers to the systematic process of defining, parsing, and persisting the components of a Data Product according to the OpenDPI specification. This system manages the registration of core entities such as Ports, Connections, and Schemas, ensuring that they are correctly resolved from various sources (inline, components, or external files) and made available for subsequent operations like translation and documentation generation.

The registration lifecycle involves taking raw YAML or JSON definitions and transforming them into a validated internal Spec structure. This structure serves as the source of truth for the CLI, allowing users to describe artifacts, manage sessions, and translate schemas into various target formats like PySpark, Protobuf, or SQL.
Sources: internal/opendpi/parse.go:27-36, CONTRIBUTING.md:46-60

Core Artifact Components

The registration system handles four primary types of artifacts that constitute a Data Product definition.

Artifact Type	Description
Ports	Define the entry/exit points for data, including descriptions and references to schemas and connections.
Connections	Configuration for data infrastructure (e.g., PostgreSQL, Kafka), including host details and variables.
Schemas	Data structure definitions using JSON Schema, which can be defined inline, as shared components, or in external files.
Tags	Metadata used to categorize and organize artifacts within the Data Product.

Sources: internal/opendpi/parse.go:210-256, internal/opendpi/parse_test.go:34-47

Artifact Parsing and Resolution

Registration begins with the Parser component, which decodes OpenDPI specifications from JSON or YAML. The parser performs a critical resolution step where it unifies disparate definition styles into a consistent internal model.

Reference Resolution Flow

Artifacts often reference each other using JSON Pointers (e.g., #/connections/db). The parser must resolve these pointers to actual objects during registration.

The parsing logic specifically handles external schema files by using a Loader to fetch files from the filesystem and resolve relative paths.
Sources: internal/opendpi/parse.go:17-25, internal/opendpi/parse.go:50-85

Schema Normalization

A significant part of artifact registration is the normalization of Schemas. The system supports:

1. Inline Schemas: Defined directly within a Port.
2. Component Schemas: Defined in the components/schemas section for reuse.
3. External Schemas: Defined in separate .yaml or .json files.

During registration, component references are rewritten from #/components/schemas/ to the standardized JSON Schema #/defs/ (or $defs) format to ensure compatibility with downstream translators.
Sources: internal/opendpi/parse.go:133-165, internal/opendpi/parse_test.go:134-150

Persistence and Writing

The Writer component handles the reverse process of registration: taking an internal Spec and persisting it back to the filesystem. This is used when the CLI modifies artifact definitions or initializes new projects.

The toRaw function is responsible for building a reverse lookup table of connections to ensure that Port connections are written back as strings (e.g., #/connections/my_db) rather than expanded objects.
Sources: internal/opendpi/writer.go:21-45, internal/opendpi/writer.go:88-100

Artifact Inspection

Once registered, artifacts can be inspected using the describe command. This utilizes the session.Context to access the parsed Spec.

Port Description Logic

The registration of a Port includes its name, description, schema details, and associated connections. The CLI provides a human-readable output (text) or machine-readable formats (JSON/YAML) for these registered artifacts.

// internal/commands/ports_describe.go:133-145
if len(port.Connections) > 0 {
    fmt.Println("Connections:")
    for _, pc := range port.Connections {
        connName := findConnectionName(pc.Connection, ctx.Spec.Connections)
        if pc.Connection != nil {
            fmt.Printf("  - %s (%s://%s) -> %s\n", connName, pc.Connection.Type, pc.Connection.Host, pc.Location)
        } else {
            fmt.Printf("  - %s -> %s\n", connName, pc.Location)
        }
    }
}

Sources: internal/commands/ports_describe.go:106-146, internal/commands/ports_describe.go:150-185

Translator Registration

The system also registers "Translators" which are components capable of converting registered JSON Schemas into target language artifacts (e.g., Java, Python, SQL). These are registered in a central map during the CLI's initialization.

Target Format	Translator Implementation
PySpark	pyspark.Translator
Protobuf	protobuf.Translator
Pydantic	pydantic.Translator
Go Types	gotypes.Translator
Databricks SQL	databrickssql.Translator

Sources: cmd/gendocs/main.go:37-52, internal/translate/databrickssql/translator.go:34-40

Conclusion

Artifact Registration serves as the foundational data management layer of the Daco CLI. By resolving complex references and normalizing schemas into a unified internal representation, it enables the tool to provide consistent artifact descriptions, reliable cross-format translations, and robust Data Product documentation.
Sources: internal/opendpi/parse.go:193-206, CONTRIBUTING.md:46-55