Schema Loader & Resolution

Relevant source files

The following files were used as context for generating this wiki page:

• internal/jschema/loader.go
• internal/jschema/traverse.go
• internal/jschema/schema.go
• internal/opendpi/parse.go
• internal/translate/prepare.go

The Schema Loader & Resolution system is a core component of the Daco CLI responsible for ingesting, parsing, and resolving JSON Schemas used within OpenDPI specifications. Its primary purpose is to transform fragmented schema definitions—which may span multiple files or contain internal references—into a unified, traversable structure suitable for translation into target programming languages or data formats.

The system handles the complexity of the JSON Schema specification, including support for $ref resolution (both internal and external), recursive traversal of schema trees, and topological sorting of definitions to ensure that dependent types are processed in the correct order.

Sources: internal/jschema/loader.go, internal/opendpi/parse.go

Core Architecture

The architecture is built around three functional layers: the File Loader, the Resolver, and the Traverser.

Component Overview

Component	Responsibility	Primary Files
Loader	Reads schema files from the filesystem and unmarshals them from JSON or YAML.	internal/jschema/loader.go
Traverser	Provides an iterator-based mechanism to walk through all nodes of a schema tree while handling circular references.	internal/jschema/traverse.go
Resolver	Replaces $ref pointers with actual schema content and rewrites internal references to a standardized $defs format.	internal/jschema/loader.go, internal/jschema/traverse.go
OpenDPI Parser	Orchestrates the loading of schemas specifically within the context of an OpenDPI specification.	internal/opendpi/parse.go

Sources: internal/jschema/loader.go:16-24, internal/jschema/traverse.go:17-21, internal/opendpi/parse.go:42-55

Schema Loading and File Resolution

The Loader struct utilizes an fs.FS abstraction to read files, ensuring compatibility with various storage backends. It supports .json, .yaml, and .yml extensions.

Loading Logic

When a schema file is loaded, the system determines the format by file extension and unmarshals the raw bytes into a jsonschema.Schema object. If the schema contains external file references (checked via IsFileRef), the ResolveRefs method recursively fetches those files and merges them into the main schema tree.

The diagram shows the recursive process of resolving file-based references into a single schema object.
Sources: internal/jschema/loader.go:27-56, internal/jschema/loader.go:61-80

Schema Traversal and Dependency Resolution

To process complex schemas, the system implements a robust traversal engine that can navigate every possible keyword in a JSON Schema (e.g., properties, allOf, items, dependentSchemas).

Traversal Engine

The Traverse function returns a Go iterator (iter.Seq). It uses a visited map to track schema nodes by pointer, which prevents infinite loops when encountering circular references.

Topological Sorting of Definitions

For code generation and translation, it is critical to process schema definitions ($defs) in an order where dependencies come before the types that use them. The topologicalSort function implements Kahn's algorithm to achieve this.

Kahn's algorithm ensures that if Schema B references Schema A, Schema A is yielded first by the iterator.
Sources: internal/jschema/traverse.go:17-133, internal/jschema/traverse.go:148-189

Integration with OpenDPI

The opendpi.Parser integrates the schema resolution logic into the broader Data Product specification. It performs several key steps:

1. Component Loading: Loads schemas defined in components/schemas.
2. Port Resolution: Identifies schemas attached to ports and resolves their external references.
3. Unified Map: Creates a Schemas map containing all resolved schemas keyed by name.
4. Reference Rewriting: Converts legacy reference formats (like #/components/schemas/ or #/definitions/) into the modern #/ $defs/ format using jschema.RewriteRefs.

Key Functions and Structures

Function / Structure	Purpose	Location
jschema.IsFileRef	Determines if a $ref string points to a local file vs. internal path.	internal/jschema/schema.go
jschema.RewriteRefs	Standardizes all internal references to use the $defs prefix.	internal/jschema/traverse.go:204-212
jschema.TraverseDefs	Iterates over $defs in topological order.	internal/jschema/traverse.go:137-149
opendpi.Spec	The final object containing fully resolved Ports, Connections, and Schemas.	internal/opendpi/parse.go:189-196

Sources: internal/opendpi/parse.go:42-146, internal/jschema/traverse.go:204-212

Schema Preparation for Templates

Before translation, the translate.Prepare function converts a resolved schema into a SchemaData structure. This step includes extracting inline nested objects into named definitions and resolving types through a language-specific TypeResolver.

func Prepare(portName string, schema *jsonschema.Schema, resolver TypeResolver) (*SchemaData, error) {
	// ... extracts key order and initializes context
	for defName, defSchema := range jschema.TraverseDefs(schema) {
		// Process $defs in topological order
		fields := ctx.resolveFields(defSchema, defPath)
		data.Defs = append(data.Defs, TypeDef{
			Name:   resolver.FormatDefName(defName),
			Fields: fields,
		})
	}
    // ...
}

Sources: internal/translate/prepare.go:28-55

Conclusion

The Schema Loader & Resolution system provides the foundation for Daco's multi-language translation capabilities. By handling the nuances of file-based schemas, circular references, and dependency ordering, it ensures that subsequent translation layers can operate on a clean, deterministic, and fully resolved representation of data structures.