Distributed Processing with Ray

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• ROADMAP.md
• configs/agents/agents_config.json
• monGARS/AGENTS.md
• scripts/generate_monGARS_llm_dataset.py
• monGARS/core/AGENTS.md

The monGARS project leverages Ray, specifically Ray Serve, to provide a robust and scalable infrastructure for distributed Large Language Model (LLM) inference. This integration is an optional but powerful feature designed for resilient deployment on workstations, edge appliances, and research clusters. When enabled, the system offloads model inference to a Ray Serve cluster, enhancing performance and enabling hardware-aware operations.

The integration is managed through a set of configuration flags and is deeply embedded into the project's deployment and observability workflows. The core application's FastAPI service issues requests directly to Ray Serve, with built-in retries and circuit-breaking logic to ensure resilience.

Sources: README.md, ROADMAP.md

Architecture and Integration

Ray Serve acts as a dedicated, distributed backend for LLM operations. The monGARS.core.llm_integration.LLMIntegration component is the primary interface, responsible for directing generation requests to either a local backend (like Ollama) or the Ray Serve HTTP endpoint, based on the USE_RAY_SERVE configuration.

This architecture allows for graceful degradation; if Ray Serve is unavailable or disabled, the system falls back to other adapters. The EvolutionEngine is also integrated with Ray, capable of notifying Ray Serve replicas about new model adapter deployments.

Sources: README.md, ROADMAP.md, monGARS/AGENTS.md, monGARS/core/AGENTS.md

Request Flow

The following diagram illustrates the conditional request flow for LLM generation.

This diagram shows how LLMIntegration routes requests to the Ray Serve cluster only when USE_RAY_SERVE is enabled.

Sources: README.md

Configuration

The Ray integration is controlled via environment variables, which are documented in the main README.md. The docker_menu.py script can also manage these settings for containerized deployments.

Environment Variables

Setting	Description	Default Value
USE_RAY_SERVE	A boolean flag to enable or disable distributed inference with Ray Serve.	false
RAY_SERVE_URL	The HTTP endpoint for the Ray Serve gateway.	http://rayserve:8000/generate
RAY_HTTP_PORT	The host port exposed for the Ray Serve HTTP gateway.	8005
RAY_DASHBOARD_PORT	The host port for the Ray Dashboard.	8265
RAY_CLIENT_PORT	The host port for the Ray Client API.	10001
RAY_MIN_WORKER_PORT	The lower bound of the port range for Ray head workers.	20000
RAY_MAX_WORKER_PORT	The upper bound of the port range for Ray head workers.	20100
RAY_HEAD_IMAGE / RAY_HEAD_IMAGE_GPU	Container image tags for the Ray head node (CPU/GPU).	Varies
RAY_SERVE_IMAGE / RAY_SERVE_IMAGE_GPU	Container image tags for the Ray Serve workers (CPU/GPU).	Varies

Sources: README.md

Deployment

Deploying the Ray cluster is streamlined through the project's Docker Compose setup and an interactive helper script. For more advanced use cases, a dedicated guide is available.

Docker Compose Deployment

The docker_menu.py script provides an interactive terminal menu to manage the Docker stack. Users can enable the "Ray profile" before startup, which automatically launches the ray-head and rayserve services.

The deployment flow using the helper script is as follows:

This process simplifies the setup of the Ray head node and the Serve HTTP gateway, making them accessible on the configured host ports. For more detailed instructions, refer to the docs/ray_serve_deployment.md document.

Sources: README.md

Observability and Monitoring

The integration with Ray includes specific observability hooks to monitor the performance and health of the distributed inference service.

• Metrics: The LLMIntegration component emits OpenTelemetry metrics for Ray Serve interactions. Key metrics include:
  • llm.ray.requests/latency
  • llm.ray.latency
  • llm.ray.failures
• Dashboard: The Ray Dashboard is exposed on the RAY_DASHBOARD_PORT and provides a web-based UI for inspecting the Ray cluster, including nodes, actors, and logs. This is the primary tool for diagnosing issues like worker restarts.

Sources: ROADMAP.md, scripts/generate_monGARS_llm_dataset.py:503-513

Operational Scenarios

The project's internal documentation outlines several operational playbooks for diagnosing and mitigating issues related to the Ray integration.

LLM Latency Spike

• Summary: This scenario addresses regressions in LLM response times.
• Diagnostics:
  • Review the llm.ray.latency histogram in your metrics backend.
  • Inspect logs from EvolutionEngine.diagnose_performance.
  • Check ModelSlotManager reconciliation logs for adapter-related issues.
• Mitigations:
  • Scale Ray workers using apply_optimizations.
  • Pre-warm model adapters.

Sources: scripts/generate_monGARS_llm_dataset.py:503-522

Worker Crash Loop

• Summary: This scenario covers situations where Ray workers are repeatedly crashing and restarting.
• Diagnostics:
  • Check the Ray Dashboard for worker restart counts.
  • Inspect logs related to ModelSlotManager resource allocation.
  • Review EvolutionEngine optimization logs.
• Mitigations:
  • Scale down or roll back problematic model adapters.
  • Increase the CPU/memory resources allocated to Ray workers.

Sources: scripts/generate_monGARS_llm_dataset.py:553-572

Conclusion

The integration of Ray Serve provides monGARS with a powerful, optional backend for distributed LLM inference. It is designed for scalability and resilience, with comprehensive configuration, deployment automation, and observability hooks. By offloading inference to a dedicated Ray cluster, monGARS can efficiently serve models in demanding, production-like environments while maintaining hardware awareness and operational flexibility.

Sources: README.md, ROADMAP.md