Sustainability & Energy Monitoring

Relevant source files

The following files were used as context for generating this wiki page:

• modules/evolution_engine/sustainability.py
• monGARS/core/sustainability_dashboard.py
• modules/evolution_engine/energy.py
• ROADMAP.md
• monGARS/core/AGENTS.md

The Sustainability & Energy Monitoring system in monGARS is designed to provide carbon-aware scheduling and energy usage reporting. Its primary purpose is to gate resource-intensive operations, such as the evolution engine's training rollouts, based on real-time sustainability telemetry. This allows the system to automate energy consumption reporting and perform hardware-aware scaling, aligning with the project's future "Phase 7 – Sustainability & Longevity" goals.

The system collects energy and reinforcement learning metrics, persists them in a central dashboard file, and uses a policy engine to evaluate whether a given operation should proceed, be delayed, or be paused. This functionality is crucial for running monGARS in an environmentally and operationally sustainable manner, especially during self-improvement and research cycles.

Sources: ROADMAP.md, modules/evolution_engine/sustainability.py, monGARS/core/sustainability_dashboard.py, monGARS/core/AGENTS.md

Architecture Overview

The sustainability monitoring architecture consists of three main parts: data collection, data aggregation/persistence, and policy evaluation.

1. Data Collection: Energy usage is captured in EnergyUsageReport objects. Other operational data, like LongHaulValidationSummary from reinforcement learning cycles, is also collected.
2. Data Aggregation: The SustainabilityDashboardBridge acts as a central hub. It receives reports, emits OpenTelemetry metrics, and persists the data into a single JSON file that serves as the "dashboard".
3. Policy Evaluation: The CarbonAwarePolicy class reads the dashboard file and applies a set of rules to make a CarbonAwareDecision on whether to proceed with an operation.

This diagram illustrates the overall data flow from data generation to policy decision.

Key Components

The system is built around a few key classes that handle data representation, persistence, and decision-making.

Component	File	Description
EnergyUsageReport	modules/evolution_engine/energy.py	A dataclass that holds detailed information about energy consumption for a specific workload.
SustainabilityDashboardBridge	monGARS/core/sustainability_dashboard.py	Persists sustainability telemetry to a JSON file and emits metrics for monitoring dashboards. It acts as the bridge between data sources and the persistent dashboard state.
CarbonAwarePolicy	modules/evolution_engine/sustainability.py	Interprets the telemetry from the dashboard to gate training rollouts and other intensive operations based on configurable sustainability rules.
CarbonAwareDecision	modules/evolution_engine/sustainability.py	A dataclass representing the outcome of a CarbonAwarePolicy evaluation, indicating whether to proceed and why.

Component Relationships

The following class diagram shows the relationships and key attributes of the main components.

Data Persistence and Metrics

SustainabilityDashboardBridge

The SustainabilityDashboardBridge is responsible for centralizing sustainability data. It is initialized with a path to a storage file (the dashboard) and limits for the number of reports to retain.

When record_energy_report or record_reinforcement_summary is called, the bridge:

1. Formats the incoming report into a JSON-serializable dictionary.
2. Emits a corresponding set of OpenTelemetry metrics.
3. Loads the existing dashboard JSON file.
4. Appends the new record to the appropriate list (energy_reports or reinforcement_runs).
5. Trims the report lists to _max_energy_reports or _max_reinforcement_records to prevent the file from growing indefinitely.
6. Writes the updated payload back to the dashboard file.

Sources: monGARS/core/sustainability_dashboard.py:83-131

OpenTelemetry Metrics

The bridge emits a rich set of metrics that can be scraped by a monitoring solution like Prometheus to build dashboards.

Metric Name	Type	Unit	Description
llm.sustainability.energy.wh	Histogram	Wh	Energy consumed by reinforcement and evolution workloads.
llm.sustainability.energy.duration_seconds	Histogram	s	Wall clock duration associated with recorded energy reports.
llm.sustainability.energy.emissions_grams	Histogram	g	Carbon emissions associated with recorded energy usage.
llm.sustainability.energy.carbon_intensity_gco2_per_kwh	Histogram	gCO2/kWh	Carbon intensity observed during recorded energy usage.
llm.sustainability.reinforcement.success_rate	Histogram		Success rate from reinforcement long-haul validation runs.
llm.sustainability.reinforcement.total_reward	Histogram		Total reward accumulated during reinforcement validation runs.
llm.sustainability.reinforcement.energy_wh	Histogram	Wh	Total energy consumed during reinforcement validation runs.
llm.sustainability.reinforcement.approval_pending	Histogram		Pending approval counts observed at the end of validation runs.
llm.sustainability.reinforcement.incident_count	Histogram		Incident counts surfaced during reinforcement validation runs.

Sources: monGARS/core/sustainability_dashboard.py:17-79

Carbon-Aware Policy Engine

The CarbonAwarePolicy class is the brain of the system, responsible for making decisions based on the collected data.

Initialization and Configuration

The policy is configured with several thresholds that dictate its behavior.

Parameter	Type	Default	Description
dashboard_path	str | Path	-	Path to the JSON dashboard file.
carbon_pause_threshold	float	550.0	Carbon intensity (gCO2/kWh) at which rollouts are paused.
carbon_caution_threshold	float	400.0	Carbon intensity (gCO2/kWh) at which rollouts are delayed.
energy_budget_wh	float	7500.0	Total energy (Wh) allowed within the energy_window.
energy_window	timedelta	12 hours	The time window over which the energy budget is calculated.
approvals_threshold	int	10	Number of pending operator approvals that will block a rollout.
cooldown	timedelta	2 hours	The recommended delay if a hard limit (pause threshold, energy budget) is exceeded.
stale_data_threshold	timedelta	6 hours	If telemetry is older than this, the policy operates more cautiously.
incident_blocks	bool	True	Whether recent reinforcement incidents should block rollouts.

Sources: modules/evolution_engine/sustainability.py:65-95

Evaluation Logic

The evaluate() method executes the core decision-making logic. The following sequence diagram shows the evaluation flow.

The evaluate method checks multiple conditions in order:

1. Stale Data: Checks if the latest telemetry data is too old.
2. Carbon Intensity: Compares the latest carbon intensity against _carbon_pause_threshold and _carbon_caution_threshold.
3. Energy Budget: Checks if the total energy consumed within the _energy_window exceeds the _energy_budget_wh.
4. Pending Approvals: Checks if the number of pending operator approvals exceeds the _approvals_threshold.
5. Incidents: Checks if there are any recent reinforcement incidents.

If any check fails, should_proceed is set to False, a reason is generated, and a recommended_delay may be calculated. If all checks pass, the decision is to proceed.

Sources: modules/evolution_engine/sustainability.py:97-203

Decision Outcome

The evaluate() method returns a CarbonAwareDecision object, which provides a structured outcome.

# Source: modules/evolution_engine/sustainability.py:13-30
@dataclass(frozen=True)
class CarbonAwareDecision:
    """Outcome of evaluating whether a rollout should proceed."""

    should_proceed: bool
    reason: str
    carbon_intensity_g_co2_per_kwh: float | None = None
    energy_window_wh: float | None = None
    approvals_pending: int | None = None
    incidents: int | None = None
    recommended_delay: timedelta | None = None
    metadata: dict[str, Any] = field(default_factory=dict)

This object allows the calling service (e.g., the EvolutionOrchestrator) to not only know whether to proceed but also why the decision was made and for how long it should potentially wait before retrying.

Conclusion

The Sustainability & Energy Monitoring system is a forward-looking feature that integrates environmental and operational awareness directly into the MLOps lifecycle of monGARS. By tracking energy usage, carbon intensity, and operational metrics, it provides an automated mechanism to govern resource-intensive tasks. This ensures that as the system self-improves through its evolution engine, it does so in a cost-effective and sustainable manner, fulfilling the long-term vision outlined in the project's roadmap.

Sources: ROADMAP.md, modules/evolution_engine/AGENTS.md, monGARS/core/AGENTS.md