Repository Tooling & Scripts

Relevant source files

The following files were used as context for generating this wiki page:

• README.md
• scripts/manage_agents.py
• scripts/update_docs_metadata.py
• scripts/generate_monGARS_llm_dataset.py
• configs/agents/agents_config.json
• tools/monGARS_deep_scan/README.md
• tests/test_extractors_unit.py
• monGARS_structure.txt

The monGARS repository includes a suite of custom scripts and tools designed to automate development workflows, enforce project standards, manage documentation, and generate datasets. These utilities are crucial for maintaining code quality, consistency, and operational efficiency. They cover areas from local environment setup and deployment orchestration to the automated generation of both project documentation and LLM training data.

The primary tooling can be categorized into development/deployment automation, documentation management, and dataset generation.

Key Scripts Overview

Script/Tool	Location	Description
Docker Menu	scripts/docker_menu.py	An interactive CLI for managing the Docker Compose development stack.
Agents Manager	scripts/manage_agents.py	Generates and maintains scoped AGENTS.md contribution guides from a central JSON configuration.
Docs Metadata Updater	scripts/update_docs_metadata.py	Automatically updates Last updated timestamps in Markdown files based on Git history.
LLM Dataset Generator	scripts/generate_monGARS_llm_dataset.py	Creates a structured LLM training dataset by analyzing the codebase, OpenAPI specs, and architectural plans.
Deep Scanner	tools/monGARS_deep_scan/	A CLI tool that inspects the repository to extract user-facing text and generate datasets for SFT, agent handoff, and embeddings.
Model Provisioner	scripts/provision_models.py	A CLI tool to download and validate LLM models required for the active profile.

Sources: README.md, monGARS_structure.txt

Development & Deployment

Docker Menu (docker_menu.py)

The docker_menu.py script provides an interactive terminal menu to simplify the management of the Docker Compose stack. It acts as a user-friendly orchestrator for common development tasks, reducing the need for manual docker compose commands and configuration management.

Key features include:

• First-Run Setup: Automatically creates a .env file from .env.example and rotates weak default secrets for JWT, Django, Postgres, and Vault.
• Port Collision Scanning: Scans for port collisions across all services (API, webapp, Postgres, Redis, etc.) and automatically assigns the next available port, persisting the changes back to .env.
• Service Orchestration: Provides menu options to build, start, stop, restart, inspect, and tail logs for the entire stack or individual services.
• Optional Profiles: Allows toggling of optional services like Ollama or Ray before startup.
• Diagnostics & Healing: Includes guided diagnostics to find and remediate issues like missing .env entries or stale ports, and an auto-heal routine to restart unhealthy services.
• Environment Summary: Displays a redacted summary of environment variables for safe debugging in shared terminals.

Sources: README.md

The following diagram illustrates the user flow through the interactive menu.

Documentation & Metadata Management

The project uses automated scripts to keep documentation consistent and up-to-date.

manage_agents.py

This script is responsible for generating and maintaining the AGENTS.md files found throughout the repository. These files serve as scoped contribution guides, defining rules and context for specific modules or directories. The script ensures that all AGENTS.md files are derived from a single source of truth, configs/agents/agents_config.json.

The generation process involves:

1. Loading the central agents_config.json file, which contains profiles for each AGENTS.md file.
2. Parsing the ROADMAP.md file to pull in relevant tasks for the "Roadmap Alignment" section of each guide.
3. Rendering each profile into a Markdown file with a standard structure, including scope, automation notes, roadmap alignment, and specific section content (bullets and paragraphs).
4. Writing the generated content to the target AGENTS.md file.

The script supports two main commands:

• refresh: Regenerates all configured AGENTS.md files.
• create: Scaffolds a new AGENTS.md file and adds its profile to the central JSON configuration.

Sources: scripts/manage_agents.py, configs/agents/agents_config.json, AGENTS.md

This diagram shows the data flow for the manage_agents.py script.

update_docs_metadata.py

This utility script maintains the Last updated timestamp present at the top of most Markdown files. It is designed to be run before committing documentation changes to ensure all timestamps are fresh.

The script works by:

1. Iterating through all Markdown files in the repository (excluding AGENTS.md and files in node_modules, .venv, etc.).
2. For each file, it uses git log to find the date of the last commit that modified that file.
3. It then inserts or updates a line near the top of the file, formatted as: > **Last updated:** YYYY-MM-DD _(auto-synced; run \python scripts/update_docs_metadata.py`)_`.

This automation ensures that readers can quickly assess the currency of any given document.

Sources: scripts/update_docs_metadata.py:1-12, 79-96

Dataset Generation

The repository contains powerful tools for generating datasets used for training and evaluating the monGARS LLM.

tools/monGARS_deep_scan/

The "monGARS Deep Scanner" is a CLI tool that inspects the repository's source code and documentation to extract user-facing text and assemble it into structured datasets.

The scanner produces the following outputs:

• sft_dataset.jsonl: For supervised fine-tuning, containing instruction-response pairs.
• agent_handoff_dataset.jsonl: For training agentic workflows, containing instructions and corresponding JSON outputs.
• embeddings_corpus.jsonl: A collection of text paragraphs for training embedding models.
• provenance.csv: A CSV file tracking the source file and line numbers for each record.
• report.md: A quality assurance report summarizing the scan.

The tool uses various extractors to parse different file types, including Python docstrings, YAML configurations, Dockerfiles, shell scripts, and HTML/JSX files.

Sources: tools/monGARS_deep_scan/README.md, tests/test_extractors_unit.py:13-75, CURATION_GUIDE.md

CLI Usage

Argument	Description	Default
--input <path>	Directory or .zip archive to scan.	-
--out <dir>	Output directory for datasets and reports.	output/
--allow-network	Toggles optional network access.	Disabled
--max-lines <N>	Skips files with more than N lines.	50,000
--jobs <N>	Number of worker processes to use.	2 * CPU (capped at 8)
--dry-run	Preview the scan without writing files.	-
--qc-terms <file>	Path to a file with Québécois French keywords for detection.	-
--include-ext <exts>	Comma-separated list of file extensions to include.	-
--exclude-dir <dirs>	Comma-separated list of directories to exclude.	-

Sources: tools/monGARS_deep_scan/README.md

scripts/generate_monGARS_llm_dataset.py

This script generates a high-quality, multifaceted dataset for training the monGARS LLM on its own architecture, operations, and API. It programmatically creates prompt-response pairs based on static analysis of the codebase and predefined architectural knowledge.

The script gathers information from several sources:

1. Python Module Analysis: It uses Python's ast module to parse all .py files in the monGARS/ and modules/ directories. It extracts docstrings, class/function names, and dependencies to generate prompts about module responsibilities, dependencies, and testing strategies.
2. Architectural Scenarios: It uses hardcoded Scenario, Playbook, and AlignmentPlan data structures to create examples related to architectural flows, operational troubleshooting, and AI safety alignment.
3. OpenAPI Specification: It can load an openapi.json file to generate prompts and responses about API endpoint usage, request/response schemas, and testing.

The final output is a monGARS_llm_dataset.jsonl file, along with split train and val sets, where each record is a JSON object containing a prompt, a response, and detailed metadata.

Sources: scripts/generate_monGARS_llm_dataset.py

The data flow for this script is visualized below.