Streaming Formats (Avro & Protobuf)
Relevant source files
The following files were used as context for generating this wiki page:
Streaming Formats in the Daco CLI project refer to the specialized translators that convert OpenDPI-based JSON schemas into industry-standard binary serialization formats: Apache Avro and Protocol Buffers (proto3). These translators allow data contracts defined in the project to be used in high-performance streaming environments like Kafka or gRPC.
The architecture relies on a shared translate.Prepare phase that normalizes JSON schemas before format-specific logic is applied. This ensures consistency across different output types while respecting the unique type systems and structural requirements of Avro and Protobuf.
Apache Avro Translation
The Avro translator converts JSON schemas into Avro Schema (.avsc) JSON documents. It maps JSON types to Avro primitives and logical types, utilizing a recursive "inline-at-first-use" strategy for definitions found in $defs.
Architecture and Data Flow
The conversion process identifies whether a field is nullable and wraps it in a union with "null" if necessary. It also handles logical types like dates and timestamps by mapping them to specialized Avro structures.
The diagram above illustrates the decision logic used by the Avro translator to map JSON schema elements to Avro-specific types.
Sources: internal/translate/avro/translator.go:61-125
Logical Type Mapping
The translator supports specific logical types to preserve semantic meaning for strings with formats:
| JSON Format | Avro Type | Logical Type |
|---|---|---|
date | int | date |
date-time | long | timestamp-millis |
uuid | string | uuid |
Sources: internal/translate/avro/translator.go:116-122, internal/translate/avro/translator_test.go:73-100
Nullability and Unions
In Avro, optional fields are represented as unions. The translator automatically detects required fields from the JSON schema; if a field is not required, it is rendered as ["null", <type>].
Sources: internal/translate/avro/translator.go:82-84, internal/translate/avro/translator_test.go:210-234
Protocol Buffers (proto3) Translation
The Protocol Buffers translator generates .proto files following the proto3 syntax. Unlike Avro, which inlines records, the Protobuf translator extracts nested objects and references into top-level message definitions to maintain idiomatic Protobuf structure.
Component Logic
The translator assigns sequential field numbers (tags) to every field within a message, starting from 1. It also handles package naming based on the output directory provided during the translation process.
This flow represents the automated tagging and templating process used to generate valid Protobuf source code.
Sources: internal/translate/protobuf/translator.go:37-64
Data Structure Mapping
Protobuf translation maps JSON types to specific proto3 scalars. Notably, all date and string formats are currently mapped to the standard string type in proto3.
| JSON Type / Format | Protobuf Type | Modifier |
|---|---|---|
string | string | - |
integer | int64 | - |
number | double | - |
boolean | bool | - |
array | (item type) | repeated |
| Optional Field | (type) | optional |
Sources: internal/translate/protobuf/translator_test.go:42-59, internal/translate/protobuf/translator_test.go:94-110, internal/translate/protobuf/translator_test.go:193-214
Schema Resolution and Components
Both translators rely on the opendpi parsing logic to resolve external references and components. Before translation occurs, the system resolves $ref pointers to ensure that all necessary metadata is available to the format-specific logic.
Reference Resolution Flow
This sequence shows how the project handles complex schemas with nested or external references before the Avro or Protobuf translation begins.
Sources: internal/opendpi/parse_test.go:124-173, internal/translate/avro/translator.go:61-64
Summary
The Streaming Formats module provides a robust bridge between high-level OpenDPI data product specifications and technical serialization formats. By automating the mapping of nullability, logical types, and complex nested relationships into Avro records and Protobuf messages, the system ensures that data contracts remain consistent across the entire data engineering lifecycle.