Data Types and Values Mapping
Relevant source files
The following files were used as context for generating this wiki page:
The Data Types and Values Mapping system in the Milvus Rust SDK provides the translation layer between Rust native types and the structured Protobuf definitions required by the Milvus server. This system ensures that collection schemas, entity data, and query parameters are correctly serialized for network transmission and deserialized for application use.
The scope of this module covers scalar types (integers, strings, booleans), vector types (dense and sparse), and complex structures like arrays and JSON. It defines how these types are categorized into fields and columns during operations like insert, query, and search.
Core Data Type Categorization
Data types in Milvus are primarily managed through the DataType enumeration. These types are divided into scalar fields for metadata and vector fields for similarity search.
Scalar and Vector Types
Milvus supports a wide range of scalar types including fixed-size integers (Int8 through Int64), floating-point numbers (Float, Double), and variable-length types (String, VarChar, JSON). Vector types are specialized for high-dimensional data, supporting binary vectors, float vectors (including FP16 and BF16), and sparse vectors.
Sources: src/proto/milvus.proto.schema.rs:727-759
Relationship Diagram
The following diagram illustrates the relationship between the CollectionSchema, its constituent FieldSchema, and the supported DataType.
The hierarchy ensures that every field in a Milvus collection is strictly typed. Sources: src/proto/milvus.proto.schema.rs:4-38, src/proto/milvus.proto.schema.rs:59-79
Value Representation and Storage
Values are encapsulated in different containers depending on whether they represent schema defaults, query placeholders, or batch data for insertion.
ValueField and ScalarField
For single values or schema defaults, the SDK uses ValueField. For batch data, ScalarField and its associated array types (e.g., LongArray, StringArray) are used to group multiple values of the same type efficiently.
Sources: src/proto/milvus.proto.schema.rs:188-212, src/proto/milvus.proto.schema.rs:213-242
Mapping Table: Rust to Milvus Types
| Rust Type | Milvus DataType | Protobuf Container |
|---|---|---|
bool | Bool | BoolArray |
i64 | Int64 | LongArray |
f32 | Float | FloatArray |
String | VarChar / String | StringArray |
Vec<f32> | FloatVector | VectorField::FloatVector |
Vec<u8> | BinaryVector | VectorField::BinaryVector |
Sources: src/proto/milvus.proto.schema.rs:727-759, examples/query_search.rs:114-129
Field Data and Columnar Organization
When performing mutations or retrieving results, data is organized into FieldData structures. This format supports columnar storage, which is optimized for high-performance processing in Milvus.
FieldData Logic Flow
The mapping from raw input to FieldData involves identifying the type and wrapping the corresponding scalar or vector array.
This flow represents the path data takes from the SDK user's input to the gRPC request. Sources: src/proto/milvus.proto.schema.rs:364-386, src/proto/milvus.proto.msg.rs:1-25
Placeholder Values
For search operations, query vectors are mapped via PlaceholderValue and PlaceholderGroup. These support a 2D-array structure where each row contains a query vector in byte format.
Sources: src/proto/milvus.proto.common.rs:44-55
Examples of Type Usage
In practice, types are defined during collection creation and used as FieldColumn during insertion.
// Defining a schema with different types
let schema = CollectionSchemaBuilder::new(COLLECTION_NAME, "description")
.add_field(FieldSchema::new_primary_int64("id", "user id", false))
.add_field(FieldSchema::new_int64("age", "age of user"))
.add_field(FieldSchema::new_float_vector("picture", "", DIM))
.build()?;
// Preparing data columns for insertion
let id_column = FieldColumn::new(schema.get_field("id").unwrap(), ids);
let picture_column = FieldColumn::new(schema.get_field("picture").unwrap(), picture_data);
Sources: examples/query_search.rs:104-112, examples/query_search.rs:125-128
The mapping allows the SDK to automatically handle the conversion of Rust vectors into the corresponding FieldData messages for the Milvus Insert API.
Sources: src/proto/milvus.proto.milvus.rs:434-445
Conclusion
The data types and values mapping in the milvus-sdk-rust project provides a robust framework for handling Milvus's multi-modal data requirements. By strictly following the Protobuf schema definitions while offering Rust-friendly abstractions like FieldColumn and CollectionSchemaBuilder, the SDK facilitates seamless interaction with the Milvus vector database.