Introduction to Daco CLI
Relevant source files
The following files were used as context for generating this wiki page:
Daco CLI is described as a "Swiss army knife for data product projects," designed to facilitate the management, definition, and translation of data schemas within a data engineering ecosystem. It provides a set of tools to initialize projects, manage OpenDPI (Open Data Product Interface) specifications, and translate these specifications into various target formats such as SQL, Python, and Scala.
The CLI is built using the Go programming language and follows a modular architecture that separates command-line interface logic from core internal packages handling configuration, parsing, and schema translation. It adheres to design principles emphasizing simplicity, idiomatic patterns, and minimal dependencies.
Sources: README.md:3, CONTRIBUTING.md:20-25, cmd/daco/main.go:1-18
Project Architecture and Structure
The Daco CLI follows a standard Go project structure, segregating the entry point from private application logic. The application is organized into a primary command-line interface (cmd/daco) and internal packages (internal/) that handle specific domain logic.
This diagram illustrates the high-level dependency flow from the entry point through the core internal modules.
Key Directories
| Directory | Purpose |
|---|---|
cmd/daco/ | CLI entry point and application bootstrap. |
internal/commands/ | Definitions for CLI commands (e.g., init, ports describe). |
internal/config/ | Project-level configuration handling for daco.yaml. |
internal/opendpi/ | Parsing and validation of OpenDPI specifications. |
internal/translate/ | Logic for converting JSON schemas to target language types. |
Sources: CONTRIBUTING.md:58-75, cmd/daco/main.go:1-18
Project Initialization and Configuration
Starting a new Daco project involves the init command, which sets up the necessary file structure and metadata. A standard project consists of a daco.yaml configuration file and a spec/ directory containing an opendpi.yaml file.
The initialization flow determines how project metadata is collected and persisted to the filesystem.
Configuration Model (daco.yaml)
The configuration is managed by the internal/config package and uses the following structure:
- Version: Integer representing the config format version (currently 1).
- Path: String pointing to the location of the OpenDPI specifications (defaults to
./spec).
Sources: internal/commands/init.go:30-45, internal/config/config.go:14-18
OpenDPI Specification Handling
Daco CLI centers around the OpenDPI specification, which defines the metadata, connections, and ports of a data product. The internal/opendpi package is responsible for parsing these specs from YAML or JSON and resolving schema references.
Specification Components
- Info: Metadata including Title, Version, and Description.
- Connections: Definitions of data sources (e.g., Host, Type, Variables).
- Ports: Access points for the data product, including descriptions, connection references, and schemas.
- Schemas: A unified map of resolved JSON schemas.
Schema Resolution Process
The parser resolves $ref links, including external file references, and organizes them into a unified structure. It rewrites references to use a standard $defs format to ensure compatibility with downstream translators.
Sources: internal/opendpi/parse.go:132-150, internal/opendpi/parse.go:189-225
Schema Translation System
One of the core features of Daco CLI is translating JSON-based schemas into language-specific structures. This system uses a "Prepare" phase to normalize the schema into a SchemaData structure before applying language templates.
The translation sequence shows how schemas are prepared and resolved into language-specific types.
Translation Support
The CLI supports various target formats through the internal/translate sub-packages:
- Python: Pydantic models, PySpark StructTypes.
- Scala: Case classes, Spark StructTypes.
- SQL: Databricks SQL (Delta Lake) CREATE TABLE statements.
- Protobuf: Proto3 message definitions.
Sources: internal/translate/prepare.go:21-45, internal/translate/databrickssql/translator.go:37-52, internal/translate/pydantic/translator.go:36-45
Summary
Daco CLI provides a structured approach to managing data product definitions by leveraging OpenDPI specifications. By automating the initialization of projects and the translation of complex JSON schemas into production-ready code for Spark, SQL, and various programming languages, it serves as a bridge between data architecture and implementation.
Sources: README.md:1-5, CONTRIBUTING.md:1-10