Command Line Interface (oxidize-cli)

Relevant source files

The following files were used as context for generating this wiki page:

• oxidize-cli/src/main.rs
• oxidize-cli/src/pipeline.rs
• oxidize-cli/AGENTS.md
• README.md
• QUICKSTART.md
• CLAUDE.md

oxidize-cli is the primary interactive interface for the oxidize LLM inference engine. It acts as a thin frontend over oxidize-core APIs, providing capabilities for single-prompt runs, interactive chat REPLs, benchmarking, and multi-stage pipeline execution. The CLI handles argument parsing, model loading via oxidize-core, and hardware-specific offload planning.

Sources: oxidize-cli/AGENTS.md:6-10, README.md:162-165

System Architecture

The CLI follows a dispatch-based architecture where a central entry point parses user input and routes it to specific execution modes. It interacts directly with oxidize-core for model management and compute backends (CPU, CUDA, Metal, MLX).

The CLI coordinates between input prompt processing, interface management, and the underlying core compute layer.
Sources: oxidize-cli/src/main.rs:172-200, README.md:204-209

Core Execution Modes

oxidize-cli supports several operational modes configured via command-line flags.

Prompt and Chat Modes

Prompt mode executes a single inference task based on a provided string. Chat mode initiates a REPL (Read-Eval-Print Loop) using std::io::BufRead, allowing for persistent conversational history during the session.

• Prompt: Triggered by the --prompt flag.
• Chat: Triggered by the --chat flag; handles Ctrl-C for graceful exit.

Sources: oxidize-cli/src/main.rs:103-118, oxidize-cli/AGENTS.md:23-28

Pipeline and Mesh Modes

Pipeline mode allows for distributed or batch processing. pipe_head tokenizes and runs initial layers before shipping hidden states to a peer, while pipe_tail listens for hidden states to complete the generation.

• Pipeline Head: --pipe-head connects to a peer via --pipe-peer.
• Pipeline Tail: --pipe-tail binds to a port via --pipe-listen.
• Mesh: Distributed inference using a libp2p mesh network, enabled via --mesh.

Sources: oxidize-cli/src/main.rs:141-155, oxidize-cli/src/pipeline.rs

Configuration and Parameters

The CLI provides extensive control over hardware offloading and sampling parameters through its argument structure Args.

Hardware and Backend Configuration

Flag	Type	Default	Description
--backend	Enum	Cpu	Target backend (Cpu, Cuda, Metal, Mlx, Vulkan)
--n-gpu-layers	usize	0	Number of layers to offload to GPU
--gpus	usize	1	Number of GPUs to use
--parallelism	String	pipeline	Multi-GPU strategy (tensor or pipeline)
--ctx-size	Option	N/A	Size of the KV context window
--threads	Option	N/A	CPU threads (defaults to physical cores)

Sources: oxidize-cli/src/main.rs:103-125, oxidize-cli/src/main.rs:215-220

Inference and Sampling Parameters

Parameter	Flag	Default	Description
Max Tokens	--max-tokens	512	Maximum tokens to generate
Temperature	--temperature	0.8	Sampling randomness (0.0-2.0)
Top-P	--top-p	None	Nucleus sampling threshold
Top-K	--top-k	None	Top-k sampling limit
KV DType	--kv-cache-dtype	F32	Data type for KV cache (F32, F16, Q8, Q4)

Sources: oxidize-cli/src/main.rs:113-138, QUICKSTART.md:38-42

Autotune System

The CLI includes an autotune system enabled by default via the --auto flag. It fingerprints the hardware environment and the specific GGUF model to automatically pick optimal knobs for threads, context size, and GPU offloading.

The TuningPlan is a pure function result containing decisions and rationales for inference configuration.
Sources: oxidize-cli/src/main.rs:361-380, CLAUDE.md:120-125

Speculative Decoding Support

oxidize-cli supports multiple speculative decoding strategies to accelerate inference by using a smaller "draft" model to predict tokens that are then verified by the "target" model.

• DFlash: Enabled via --draft-model. Requires the OX_DFLASH_SPECULATIVE environment variable to be active.
• EAGLE3: Supports SafeTensors or GGUF draft models for speculative generation.
• MTP (Medusa/Next-n): Uses native in-GGUF prediction layers if present.
• QuantSpec: Self-speculative decoding using hierarchical I8 TurboQuant MTP drafts.

Sources: oxidize-cli/src/main.rs:470-520, oxidize-cli/src/main.rs:645-660

Diagnostic and Profiling Tools

Diagnostic capabilities are integrated directly into the CLI and standalone binaries.

• Inspection: The inspect_gguf tool (or inspect subcommand) prints GGUF metadata and architecture.
• Benchmarking: The bench subcommand measures tokens per second (TPS) and prefill speed across multiple iterations.
• Profiling: Supports perf and samply profilers via the --profile flag. This uses OXIDIZE_PROFILE_CHILD to handle subprocess profiling.

Sources: oxidize-cli/src/main.rs:310-320, oxidize-cli/AGENTS.md:13-18, QUICKSTART.md:44-46

Summary

oxidize-cli serves as the primary entry point for local LLM operations, bridging user requirements with high-performance Rust compute backends. Through its autotune system and speculative decoding support, it provides an optimized out-of-the-box experience while remaining highly configurable for specific hardware deployments. Key responsibilities include model resolution, memory-mapped I/O management, and coordinating distributed pipeline execution.

Sources: README.md:204-213, oxidize-cli/AGENTS.md:6-10