Arrow & Parquet Support

Relevant source files

The following files were used as context for generating this wiki page:

• sea-orm-arrow/Cargo.toml
• changelog/2.0.0-rc.34.md
• changelog/2.0.0-rc.35.md
• examples/parquet_example/src/main.rs
• src/lib.rs
• sea-orm-sync/src/lib.rs
• sea-orm-sync/Cargo.toml

SeaORM provides native integration with the Apache Arrow ecosystem, allowing developers to bridge the gap between relational databases and columnar data formats. This support is primarily facilitated through the sea-orm-arrow crate, which enables the conversion of SeaORM entities and models into Arrow schemas and data structures. This integration is essential for high-performance data analytics, machine learning pipelines, and efficient data storage using Parquet.

Sources: src/lib.rs:360, sea-orm-arrow/Cargo.toml, changelog/2.0.0-rc.34.md:10-18

Core Components and Integration

The Arrow support is encapsulated in a dedicated crate, sea-orm-arrow, which is integrated into the main SeaORM ecosystem via feature flags. It provides macros and traits to automate the mapping between Rust structs (Entities) and Arrow memory structures.

Key Integration Features

• sea-orm-arrow Crate: A standalone crate for entity-to-Arrow conversion logic.
• DeriveArrowSchema: A procedural macro that generates Arrow schema definitions from SeaORM entity models.
• Data Conversion: Methods to transform ActiveModel collections into Arrow RecordBatch objects and vice versa.
• Sync Support: Integration is also available in sea-orm-sync for non-async environments.

Sources: changelog/2.0.0-rc.34.md:10-18, src/lib.rs:434, sea-orm-sync/src/lib.rs:434

Architecture Overview

The following diagram illustrates the flow of data between a relational database, SeaORM models, and the Arrow/Parquet ecosystem.

The diagram shows how SeaORM acts as the intermediary between persistent SQL storage and columnar Parquet storage using Arrow as the in-memory exchange format.
Sources: examples/parquet_example/src/main.rs:56-96, changelog/2.0.0-rc.34.md:13

Data Mapping and Type Support

SeaORM handles complex mappings between SQL types and Arrow's strongly typed columnar format. This includes specialized support for high-precision decimals and time-zone-aware timestamps.

SQL/SeaORM Type	Arrow Mapping	Notes
Decimal(p <= 18)	Decimal64	Optimized for smaller precision.
Decimal(19 <= p <= 38)	Decimal128	Standard high precision.
Decimal(p > 38)	Decimal256	Ultra-high precision.
DateTimeUtc	Timestamp	Supports various resolutions and timezones.
Binary / Bytes	BinaryArray	Supports FixedSizeBinary via attributes.

Sources: changelog/2.0.0-rc.35.md:58-63, changelog/2.0.0-rc.34.md:14

Specialized Attributes

Developers can use attributes to fine-tune the Arrow conversion:

• arrow_schema: Placed on the model struct to enable schema generation.
• arrow_byte_width = N: Used for FixedSizeBinary(N) mapping.

Sources: examples/parquet_example/src/main.rs:16, changelog/2.0.0-rc.35.md:62

Implementation Example: Parquet Roundtrip

The integration allows for a seamless "roundtrip" where data is fetched from a database (or generated), converted to Arrow, saved to Parquet, read back, and re-inserted into a database.

Workflow Sequence

This sequence ensures data integrity across different storage formats using Arrow's standardized memory layout.
Sources: examples/parquet_example/src/main.rs:56-96

Code Snippet: Model Definition

#[sea_orm::model]
#[derive(Clone, Debug, PartialEq, DeriveEntityModel)]
#[sea_orm(table_name = "measurement", arrow_schema)]
pub struct Model {
    #[sea_orm(primary_key)]
    pub id: i32,
    pub recorded_at: ChronoDateTimeUtc,
    pub temperature: f64,
    #[sea_orm(column_type = "Decimal(Some((10, 4)))")]
    pub voltage: Decimal,
}

Sources: examples/parquet_example/src/main.rs:13-25

Dependency Configuration

To use Arrow and Parquet support, specific features must be enabled in Cargo.toml.

Feature	Crate	Description
with-arrow	sea-orm / sea-orm-sync	Enables the sea-orm-arrow integration.
prettyprint	sea-orm-arrow	Enables Arrow's pretty printing capabilities.
with-rust_decimal	sea-orm-arrow	Required for Decimal type mapping.
with-chrono	sea-orm-arrow	Required for Chrono-based timestamp mapping.

Sources: sea-orm-arrow/Cargo.toml:16-21, sea-orm-sync/Cargo.toml:132

Summary

The Arrow & Parquet support in SeaORM 2.0 provides a robust mechanism for data interoperability. By leveraging the sea-orm-arrow crate and the DeriveArrowSchema macro, developers can efficiently move data between relational databases and high-performance columnar formats. This system handles the complexities of type mapping, precision management, and serialization, making it an ideal choice for data-intensive Rust applications.

Sources: changelog/2.0.0-rc.34.md:10-18, examples/parquet_example/src/main.rs:114-118