Distributed Mesh Architecture

Relevant source files

The following files were used as context for generating this wiki page:

• oxidize-core/src/mesh/mod.rs
• oxidize-core/src/mesh/node.rs
• oxidize-core/src/mesh/sharding.rs
• oxidize-core/src/mesh/AGENTS.md
• oxidize-core/src/mesh/discovery.rs
• oxidize-core/src/mesh/k8s.rs

The Distributed Mesh Architecture enables multi-node LLM inference through a libp2p-based network. It provides infrastructure for deterministic leader election, model sharding, fault tolerance, and peer discovery. This system allows the project to execute inference across a cluster of nodes, supporting models that exceed the memory capacity of a single device through pipeline or tensor parallelism.

The mesh implementation ensures namespace isolation, preventing cross-namespace peer communication, and utilizes a deterministic Bully election algorithm to maintain cluster leadership. Data flows through namespaced GossipSub topics, while a discovery service manages peer identities and capabilities.

Sources: oxidize-core/src/mesh/AGENTS.md:1-12, oxidize-core/src/mesh/discovery.rs:1-10

Core Components and Structure

The mesh architecture is organized into several specialized submodules gated by a privacy boundary in mod.rs.

Component Summary

Component	Responsibility
MeshNode	Represents a single node with specific hardware capabilities and roles.
DiscoveryService	Manages mDNS peer discovery and namespace-isolated payloads.
BullyElection	Executes deterministic leader election based on clock, seniority, and peer ID.
GossipRouter	Handles message routing and invalidates sessions to drop stale events.
ShardPlan	Defines how LLM layers are distributed across available mesh nodes.
TopologyGraph	Tracks peer health and evicts stale nodes based on heartbeat timeouts.

Sources: oxidize-core/src/mesh/AGENTS.md:8-21, oxidize-core/src/mesh/discovery.rs:60-75

Node Configuration and Capabilities

Each node in the mesh reports its NodeCapabilities to the cluster. This includes hardware-specific information such as device type (CPU or CUDA), available memory, and CPU thread count.

pub struct NodeCapabilities {
    pub device_type: String,
    pub memory_bytes: u64,
    pub cpu_threads: usize,
    pub can_shard: bool,
    pub tags: HashMap<String, String>,
}

Sources: oxidize-core/src/mesh/k8s.rs:141-165, oxidize-core/src/mesh/discovery.rs:36-40

Peer Discovery and Networking

The mesh uses libp2p with a transport stack consisting of TCP, Noise encryption, and Yamux multiplexing. Discovery is performed via mDNS, using namespaced payloads to ensure nodes only connect to peers within their designated environment.

Discovery Workflow

1. Identity Generation: Each node generates an Ed25519 keypair and a unique PeerId.
2. Payload Advertisement: Nodes advertise a DiscoveryPayload containing their namespace and hardware capabilities.
3. Namespace Filtering: Nodes accept peers only if the namespace field in the discovery payload matches their local configuration.
4. Swarm Integration: Discovered peers are added to the libp2p swarm for GossipSub communication.

Deterministic Leader Election

Leadership is determined through a Bully Election algorithm. The process is entirely deterministic to ensure all nodes independently converge on the same leader without network randomness.

Election Criteria

The winner of an election is the node with the maximum values in the following priority order:

1. election_clock (Highest value)
2. seniority
3. commands_seen
4. peer_id (Tie-breaker)

Session Invalidation

Every message in the mesh carries an election_clock. When a new election concludes, the GossipRouter invalidates the previous session. Any messages arriving with a stale clock are dropped to prevent race conditions or inconsistent states.

Sources: oxidize-core/src/mesh/AGENTS.md:29-33, oxidize-core/src/mesh/discovery.rs:250-275

Model Sharding and Parallelism

The ShardPlan determines how model weights and compute tasks are distributed across the mesh. The master node computes this plan based on the cluster TopologyGraph and broadcasts it to all workers.

Parallelism Strategies

• Pipeline Parallelism: Distributes sequential layers of the model across different nodes.
• Tensor Parallelism: Splits individual tensor operations across multiple nodes (sharding over a ring).

Sharding Data Flow

Fault Tolerance and Heartbeats

The mesh maintains cluster health through a combination of heartbeats and timeout-gated collectives.

1. Stale Peer Eviction: The TopologyGraph tracks the last seen time for every peer. If a node fails to provide a heartbeat within the timeout period (default 15 seconds), it is evicted from the topology.
2. Collective Timeouts: Distributed operations (like all_sum or all_gather) have a default timeout of 60 seconds.
3. Automatic Re-sharding: When the master detects a topology change (either a new node joining or an existing node being evicted), it automatically triggers a re-computation and broadcast of the ShardPlan.

Sources: oxidize-core/src/mesh/AGENTS.md:18-24, oxidize-core/src/mesh/discovery.rs:320-350

Kubernetes Integration

The architecture includes a specialized planning layer for Kubernetes (k8s.rs). It translates an OxidizeClusterSpec into a mesh configuration, including GPU placement requirements and RDMA fabric needs.

Cluster Specification Fields

Field	Description
model	Source model ID, format, and quantization type.
serving	Min/max replicas and API capability flags.
mesh	Namespace, parallelism strategy, and collective timeouts.
gpu	Resource name (e.g., nvidia.com/gpu) and memory requirements.
rollout	Policies for max unavailable and max surge replicas.

Sources: oxidize-core/src/mesh/k8s.rs:37-65, oxidize-core/src/mesh/k8s.rs:104-125

The Distributed Mesh Architecture serves as the foundation for scaling LLM inference within the Oxidize ecosystem. By combining deterministic leadership with automated sharding and discovery, the system maintains a robust, fault-tolerant environment for large-scale model execution.