Milvus Vector DB
Relevant source files
The following files were used as context for generating this wiki page:
Milvus Vector DB is integrated into the FLEXT ecosystem as a specialized infrastructure service designed to handle vector similarity searches and high-performance embedding storage. Within the FLEXT data integration platform, it serves as a core component for modern AI-driven workloads, enabling the project to manage enterprise data integration that requires vector-based retrieval.
The implementation is centralized within the FLEXT Docker infrastructure, ensuring that Milvus and its dependencies (etcd and MinIO) follow the project's strict Docker infrastructure standards. This setup provides a standalone vector database environment that integrates with the broader platform services like flext-core, flext-infra, and various domain packages.
Sources: docker/docker-compose.milvus.yml:1-45, README.md:1-20
Infrastructure Architecture
The Milvus deployment in FLEXT uses a standalone architecture supported by three primary containers. This setup ensures high availability for metadata management and object storage, which are prerequisite for Milvus operations.
Service Dependencies
Milvus relies on a specific sequence of service availability:
- Etcd: Used for metadata storage and service discovery.
- MinIO: Serves as the object storage layer for large-scale vector data and index files.
- Milvus Standalone: The main vector database engine that coordinates search and storage operations.
The diagram shows the dependency flow where Milvus Standalone requires etcd and MinIO to be operational before it can handle external requests.
Sources: docker/docker-compose.milvus.yml:31-45
Component Configuration
The services are configured using standard Docker Compose specifications, utilizing a dedicated bridge network named milvus.
Service Definitions
| Service | Image | Key Responsibilities | Port Mapping |
|---|---|---|---|
etcd | quay.io/coreos/etcd:v3.5.5 | Metadata storage, compaction, and revision control. | 2379 (Internal) |
minio | minio/minio:RELEASE... | S3-compatible object storage for vector data. | 9000 (API), 9001 (Console) |
milvus | milvusdb/milvus:v2.4.0 | Main vector engine performing similarity search. | 19531 (API), 9092 (Metrics) |
Sources: docker/docker-compose.milvus.yml:2-45
Storage and Persistence
Persistence is managed through three distinct Docker volumes, ensuring data survives container restarts:
etcd_data: Persists etcd revisions and metadata.minio_data: Persists the raw object storage used by Milvus.milvus_data: Persists Milvus-specific local library files in/var/lib/milvus.
Sources: docker/docker-compose.milvus.yml:58-62
Docker Infrastructure Integration
Milvus follows the FLEXT Canonical Engineering Law, which mandates centralized Docker management. This means the docker-compose.milvus.yml file is located in the ~/flext/docker/ directory, and local project-level Docker files are prohibited.
Lifecycle Management
As an infrastructure service, Milvus is managed via the tk (toolkit) API within the flext-tests and flext-infra packages. This provides:
- Test Isolation: Clean states for vector search tests.
- Dirty State Management: Automatic recreation if the vector index becomes corrupted.
- Persistent State Location: Tracking through
~/.flext/docker_state.json.
Sources: docker/README.md:100-130, AGENTS.md:50-80
Environment Configuration
The Milvus stack utilizes specific environment variables to link the internal components together:
# Extract from Milvus configuration
milvus:
environment:
ETCD_ENDPOINTS: etcd:2379
MINIO_ADDRESS: minio:9000
Sources: docker/docker-compose.milvus.yml:38-40
Operational Specifications
Connectivity
External FLEXT packages (such as flext-core or flext-api) connect to Milvus using the following mapped ports:
- Vector API:
19531(mapped from internal19530) - Web UI/Health:
9092(mapped from internal9091)
Health Checks
The stack implements automated health monitoring to ensure reliability within the data integration pipeline:
- Etcd: Verified via
etcdctl endpoint health. - MinIO: Verified via a curl request to
/minio/health/live. - Milvus: Monitored via the
/healthzendpoint on port 9091.
Sources: docker/docker-compose.milvus.yml:10-15, 25-28, 41-45
Conclusion
The Milvus Vector DB integration provides the FLEXT workspace with advanced vector search capabilities required for modern enterprise data integration. By adhering to the centralized Docker infrastructure and strict architectural governance defined in AGENTS.md, Milvus operates as a reliable, high-performance service that complements the core integration features of the platform.