Hardware-Aware Operations

Relevant source files

The following files were used as context for generating this wiki page: - [README.md](https://github.com/ales27pm/monGARS/blob/HEAD/README.md) - [ROADMAP.md](https://github.com/ales27pm/monGARS/blob/HEAD/ROADMAP.md) - [configs/agents/agents_config.json](https://github.com/ales27pm/monGARS/blob/HEAD/configs/agents/agents_config.json) - [AGENTS.md](https://github.com/ales27pm/monGARS/blob/HEAD/AGENTS.md) - [monGARS/core/AGENTS.md](https://github.com/ales27pm/monGARS/blob/HEAD/monGARS/core/AGENTS.md) - [modules/AGENTS.md](https://github.com/ales27pm/monGARS/blob/HEAD/modules/AGENTS.md)

Hardware-Aware Operations in monGARS refer to a set of features and design principles that enable the system to run efficiently on a diverse range of hardware, from low-power edge devices like Raspberry Pi to high-performance GPU clusters. This capability is central to the project's goal of being a resilient, privacy-first AI assistant that can be deployed anywhere.

The system includes mechanisms for automatic performance tuning, multi-architecture container builds, GPU-accelerated workloads, and distributed inference. These operations ensure that monGARS can adapt its resource consumption and performance characteristics to the underlying hardware, providing a consistent experience across different deployment environments. Future work aims to expand these capabilities with automated energy usage reporting and advanced hardware-aware scaling.

Sources: README.md, ROADMAP.md

Roadmap and Project Goals

Hardware-awareness is a key theme in the project's roadmap, with specific features outlined in completed and future phases. These goals are also reflected in the project's delivery charters (AGENTS.md files), which guide development.

Sources: ROADMAP.md, configs/agents/agents_config.json

Hardware-Related Roadmap Phases

Phase	Status	Key Features
Phase 3 – Hardware & Performance	✅ Completed Q3 2025	- Worker auto-tuning for Pi/Jetson (recommended_worker_count). - Multi-architecture build scripts. - Ray Serve HTTP integration with circuit breaking. - OpenTelemetry metrics for Ray Serve (llm.ray.*).
Phase 7 – Sustainability & Longevity	🌱 Future	- Automate energy usage reporting via an energy tracker pipeline. - Advanced hardware-aware scaling. - Share optimisation artefacts between nodes.

Sources: ROADMAP.md, configs/agents/agents_config.json

Core Components

Worker Auto-Tuning

To optimize performance on constrained hardware, monGARS implements an automatic worker tuning mechanism. The function monGARS.utils.hardware.recommended_worker_count() automatically adjusts the number of Uvicorn workers when the application is running on devices like a Raspberry Pi or Jetson. This avoids the need for manual configuration on these specific edge platforms.

Sources: README.md:200-202, ROADMAP.md:30

The conceptual logic for this auto-tuning is as follows:

This diagram illustrates the decision process for setting the worker count at startup.

Containerization and Multi-Architecture Builds

The project provides extensive support for containerization, with scripts and configurations tailored for different hardware architectures and use cases, including CPU, GPU, and embedded devices.

Sources: README.md:203-205, README.md:124-131

Key Features:

• Build Scripts:
  • build_native.sh: Produces x86_64 images.
  • build_embedded.sh: Produces multi-architecture images for devices like Raspberry Pi.
• Dockerfiles:
  • Dockerfile: For standard builds.
  • Dockerfile.embedded: For embedded device builds.
• GPU Support: A GPU overlay for Docker Compose is available, which uses a different base image (PYTORCH_IMAGE_GPU) to enable CUDA support for GPU workloads. This produces a distinct image tagged as MONGARS_GPU_IMAGE to avoid conflicts with CPU builds.
• Base Image Configuration: The base PyTorch image can be specified at build time using PYTORCH_IMAGE (for CPU) and PYTORCH_IMAGE_GPU (for GPU) build arguments, allowing alignment with specific CUDA driver versions without modifying the Dockerfiles.

Sources: README.md:124-131

The following diagram shows the different build paths available:

This diagram illustrates how different scripts and configurations produce architecture-specific container images.

Distributed Inference with Ray Serve

For scalable and distributed model inference, monGARS integrates with Ray Serve. This is an optional feature that can be enabled to offload LLM requests to a dedicated Ray cluster, complete with circuit breaking and performance monitoring.

Sources: README.md:144-145, ROADMAP.md:33, monGARS/AGENTS.md

• Activation: Enabled by setting the USE_RAY_SERVE configuration key to true.
• Endpoint: The Ray Serve HTTP endpoint is configured via RAY_SERVE_URL.
• Resilience: The integration includes circuit breaking to handle failures gracefully.
• Observability: Success and failure counters are exposed as OpenTelemetry metrics with the prefix llm.ray.*.

The sequence of a request using Ray Serve is shown below.

This diagram shows how FastAPI acts as a proxy to a Ray Serve cluster for LLM inference when enabled.

Configuration

Several environment variables are available to control hardware-aware features.

Setting	Description	Source
USE_RAY_SERVE	Boolean to enable distributed inference via Ray Serve.	README.md:144
RAY_SERVE_URL	The HTTP endpoint for the Ray Serve cluster.	README.md:144
MONGARS_IMAGE	Tag applied to the CPU application build.	README.md:128
MONGARS_GPU_IMAGE	Tag applied to the GPU application build.	README.md:128
PYTORCH_IMAGE	Base image for CPU builds.	README.md:129
PYTORCH_IMAGE_GPU	Base image for GPU builds, e.g., pytorch/pytorch:2.6.0-cuda12.4-cudnn9-runtime.	README.md:129
RAY_HEAD_IMAGE / RAY_SERVE_IMAGE	Container tags for Ray head and Serve nodes (CPU/GPU variants available).	README.md:130
RAY_HTTP_PORT	Host port for the Ray Serve HTTP gateway.	README.md:140
WORKER_DEPLOYMENT_NAME	Kubernetes deployment targeted by the evolution engine for scaling.	README.md:154

Sources: README.md:128-155

Future Work: Sustainability

The project roadmap includes a "Sustainability & Longevity" phase focused on enhancing hardware-aware operations with energy efficiency in mind. The plan is to automate energy usage reporting and implement advanced, hardware-aware scaling. This will leverage an "energy tracker pipeline" and reinforcement observability feeds to make intelligent decisions about resource allocation, further solidifying monGARS's ability to run responsibly in any environment.

Sources: ROADMAP.md:67-71, modules/AGENTS.md