Spark & Databricks Integration

Relevant source files

The following files were used as context for generating this wiki page:

• internal/translate/pyspark/translator.go
• internal/translate/databrickspyspark/translator.go
• internal/translate/databrickspyspark/resolver.go
• internal/translate/databricksscala/translator.go
• internal/translate/sparksql/translator.go
• internal/translate/sparkscala/translator.go
• internal/translate/databrickssql/translator.go

The Spark and Databricks integration within the Daco CLI provides a robust framework for translating JSON schemas into native big data type definitions and Data Definition Language (DDL) statements. This module enables users to generate consistent schema structures for PySpark, Spark Scala, Spark SQL, and their Databricks-specific variants, ensuring that data contracts are accurately enforced across different environments and programming languages.

Sources: internal/translate/pyspark/translator.go, internal/translate/sparksql/translator.go, internal/translate/sparkscala/translator.go

Translation Architecture

The integration follows a structured translation pattern where JSON schemas are first prepared into a neutral intermediate format and then passed to language-specific templates. The system differentiates between standard Spark implementations and Databricks-enhanced versions, which often include additional metadata or specific table formats like Delta.

Core Translation Logic

The following diagram illustrates the common flow of translating a schema into a Spark-compatible format.

The Translate method in each package utilizes translate.Prepare to resolve references and organize schema fields before rendering them through embedded .go.tmpl files.

Sources: internal/translate/pyspark/translator.go:30-34, internal/translate/sparksql/translator.go:39-43, internal/translate/sparkscala/translator.go:33-37

Supported Target Formats

The integration supports multiple output formats, summarized in the table below:

Target	File Extension	Output Description
PySpark	.py	StructType definitions using pyspark.sql.types.
Databricks PySpark	.py	StructType definitions with additional metadata comments.
Spark Scala	.scala	StructType definitions wrapped in a Scala Serializable object.
Databricks Scala	.scala	StructType with metadata and Databricks-specific comments.
Spark SQL	.sql	CREATE TABLE DDL with inlined STRUCT types.
Databricks SQL	.sql	CREATE TABLE DDL using USING DELTA.

Sources: internal/translate/pyspark/translator.go:25, internal/translate/databrickspyspark/translator.go:25, internal/translate/sparkscala/translator.go:28, internal/translate/sparksql/translator.go:34, internal/translate/databrickssql/translator.go:37

Data Type Resolution

The system maps standard JSON schema types to Spark types via specific resolver implementations.

Primitive Mapping

For Databricks PySpark, the mapping is handled by the resolver in internal/translate/databrickspyspark/resolver.go.

JSON Type	Format	Spark Type
string	date	T.DateType()
string	date-time	T.TimestampType()
string	N/A	T.StringType()
integer	N/A	T.LongType()
number	N/A	T.DoubleType()
boolean	N/A	T.BooleanType()

Sources: internal/translate/databrickspyspark/resolver.go:19-45

Complex Types and Inlining

For SQL-based translators, nested $ref types are often inlined as STRUCT<...> definitions rather than maintaining separate table references. This is achieved through the inlineStruct and renderStruct functions.

func renderStruct(fields []translate.Field) string {
	parts := make([]string, len(fields))
	for i, f := range fields {
		parts[i] = f.Name + ": " + f.Type
		if f.Tag != "" {
			parts[i] += f.Tag
		}
	}
	return "STRUCT<" + strings.Join(parts, ", ") + ">"
}

Sources: internal/translate/sparksql/translator.go:82-91, internal/translate/databrickssql/translator.go:73-82

Databricks Enhancements

Databricks-specific translators include specialized logic for metadata and table features.

Metadata and Comments

The Databricks PySpark resolver enriches fields by converting JSON schema descriptions into Spark field metadata.

func (r *resolver) EnrichField(f *translate.Field) {
	if f.Description != "" {
		escaped := strconv.Quote(f.Description)
		f.Tag = `, metadata={"comment": ` + escaped + `}`
	}
}

Sources: internal/translate/databrickspyspark/resolver.go:62-67

Delta Lake Support

The databrickssql translator is specifically designed to generate CREATE TABLE statements that utilize the Delta format, identified by the USING DELTA clause in the underlying templates.

Sources: internal/translate/databrickssql/translator.go:37

Logic for Scala Integration

In Spark Scala and Databricks Scala, the generated code is encapsulated within a Scala object to ensure it is Serializable, allowing it to be used effectively within Spark distributed jobs.

The Translate function explicitly sets an ObjectName in the extra data map using the PascalCase version of the port name.
Sources: internal/translate/sparkscala/translator.go:42, internal/translate/databricksscala/translator.go:42

Summary

The Spark & Databricks Integration module serves as a bridge between abstract JSON schema definitions and concrete big data implementations. By utilizing a common preparation layer combined with specialized resolvers and templates, Daco CLI ensures that schemas generated for PySpark, Scala, and SQL are consistent, metadata-rich, and optimized for both standard Spark and Databricks Delta Lake environments.