Project Introduction & Quickstart

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• QUICKSTART.md
• HOW_TO_INSTALL.md
• PRD.md
• CLAUDE.md
• CONTRIBUTING.md
• oxidize-core/src/lib.rs

Oxidize is a high-performance, Rust-based workspace providing local-first Large Language Model (LLM) inference and tooling. It enables running quantized models on commodity hardware, including CPUs, GPUs, and Apple Silicon, while maintaining a dependency-light footprint. Sources: README.md:1-7, PRD.md:5-13

The project organizes its capabilities into a layered architecture. This structure ensures that inference logic remains centralized in a core engine while multiple interfaces, such as a Command Line Interface (CLI), an OpenAI-compatible HTTP server, and Python bindings, share consistent behavior. Sources: README.md:237-251, CLAUDE.md:8-12

Core Architecture

Oxidize uses a layered workspace design to separate compute logic from user interfaces.

Architectural Layers

• Compute Layer (oxidize-core): Manages GGUF parsing, tensor operations, quantization primitives, and backend-specific execution (CUDA, Metal, CPU, WASM).
• Interface Layer: Provides user access via oxidize-cli, oxidize-server (HTTP API), and oxidize-py (Python bindings).
• Utility Layer: Contains oxidize-quantize for offline model weight conversion.
• Terminal UI (oxidize-tui): A standalone client that communicates with the server via HTTP/SSE.

Sources: README.md:237-245, PRD.md:20-33

Request Flow

When you provide a prompt, the system executes the following flow:

1. The interface crate receives the input.
2. oxidize-core sets up the model and session.
3. The engine performs token generation and sampling.
4. The system streams or buffers output back to you.

Sources: README.md:247-248

Sources: README.md:247-248, PRD.md:20-33

Installation and Setup

Preconditions

Before building, you must ensure the following tools are available:

• Rust Toolchain: Edition 2024 support is required.
• Make: Used for workspace-wide build commands.
• Git: Required for cloning the repository.
• wasm-bindgen-cli: Optional, required only for WASM builds.

Sources: HOW_TO_INSTALL.md:5-20, QUICKSTART.md:5-9

Build Process

You can build the workspace from source or install the binary globally.

1. Clone and Build:

   git clone https://github.com/Zapdev-labs/oxidize.git
   cd oxidize
   make build
   

   Sources: HOW_TO_INSTALL.md:32-37, QUICKSTART.md:11-15

2. Global Installation:

   cargo install --git https://github.com/Zapdev-labs/oxidize.git oxidize-cli --bin oxidize --locked
   

   Sources: HOW_TO_INSTALL.md:65-67

Quickstart Usage

CLI Inference

You can run a single prompt or enter an interactive chat mode using the CLI.

Mode	Command	Description
Single Prompt	oxidize --prompt "Hello"	Runs a one-shot inference.
Chat Mode	oxidize --chat	Starts an interactive REPL session.
With Model	oxidize --model path.gguf	Loads a specific GGUF model file.
GPU Offload	oxidize --n-gpu-layers 20	Offloads specific layers to the GPU.

Sources: README.md:83-100, QUICKSTART.md:19-35

HTTP Server

Start an OpenAI-compatible API server to handle remote requests.

cargo run -p oxidize-server -- --host 127.0.0.1 --port 8080

• Health Check: GET /healthz
• OpenAPI Specs: GET /openapi.json

Sources: README.md:112-123, QUICKSTART.md:63-71

Development Workflow

The workspace includes standard make targets for quality control and builds.

Target	Action
make fmt	Checks Rust code formatting.
make lint	Runs Clippy with warnings denied.
make test	Executes all workspace tests.
make check	Runs formatting, linting, and tests.
make wasm	Builds oxidize-core for WebAssembly.
make ci	Performs a full check and release build.

Sources: QUICKSTART.md:78-87, CONTRIBUTING.md:18-29

Perfomance Gating

Performance-sensitive changes, such as kernel updates or quantization modifications, require benchmark evidence. You should run Criterion benchmarks locally before submitting changes.

cargo bench -p oxidize-core --bench criterion -- --noplot

Sources: CONTRIBUTING.md:60-65

Quantization Guide

Quantization reduces model size and memory pressure. The utility oxidize-quantize handles these conversions.

1. Choose Target: Select a target bits-per-weight (e.g., Q4_0, Q8_0, F16).
2. Convert: Run the utility against a floating-point source.
3. Benchmark: Compare inference quality and speed against the original model.

cargo run -p oxidize-quantize -- \
  --input /models/model-f32.bin \
  --output /models/model-q4_0.bin \
  --source F32 \
  --target Q4_0

Sources: README.md:133-157, QUICKSTART.md:73-77

Summary

Oxidize provides a robust Rust ecosystem for local LLM inference. By centralizing logic in oxidize-core and offering various interface crates, it allows developers to deploy consistent inference behavior across CLI, server, and Python environments. The project emphasizes performance through SIMD-optimized CPU kernels, GPU acceleration (CUDA/Metal), and flexible quantization schemes. Sources: README.md:247-251, PRD.md:120-131