Dataset Management and Curation

Relevant source files

The following files were used as context for generating this wiki page:

• CURATION_GUIDE.md
• datasets/monGARS_llm/README.md
• monGARS/mlops/dataset.py
• models/datasets/catalog.py
• scripts/auto_unsloth_llm2vec.py
• scripts/dataset_prep.py
• scripts/generate_monGARS_llm_dataset.py
• tools/monGARS_deep_scan/README.md

Dataset management and curation are central to the monGARS project's self-improvement capabilities. The system includes a suite of tools and scripts for generating, processing, curating, and versioning datasets used for fine-tuning Large Language Models (LLMs). These datasets are derived from the repository's own source code, documentation, and predefined architectural patterns, enabling the model to reason about its own structure and behavior.

The overall workflow involves generating raw datasets from various sources, curating them for quality and relevance, processing them into a standardized format, and consuming them in training pipelines. This structured approach ensures that fine-tuning runs are reproducible and that the resulting models are well-aligned with the project's architecture and operational needs.

Dataset Generation

Multiple tools exist to generate datasets from different sources, each tailored for a specific purpose.

monGARS Deep Scanner

The monGARS_deep_scan tool is a command-line utility that inspects a given repository to extract user-facing artifacts and assemble them into structured datasets. It is the primary source for raw data that feeds into the manual curation process.

The scanner produces several output files in JSONL format, along with a provenance report for traceability.

Outputs of the Deep Scanner

File	Description
sft_dataset.jsonl	Supervised Fine-Tuning (SFT) dataset containing instruction-response pairs.
agent_handoff_dataset.jsonl	Dataset for agent workflow steps, Docker actions, and other structured outputs.
embeddings_corpus.jsonl	A corpus of text paragraphs for training embedding models.
provenance.csv	A CSV file mapping each dataset record to its source file and line range.
report.md	A QA report summarizing dataset totals, top source files, and warnings.
logs/scan.log	A log file of the scan process.

Each record in the generated datasets includes _meta provenance fields, such as source_file, line_range, and a qc_fr_ca flag for Québécois French detection.

Sources: tools/monGARS_deep_scan/README.md, CURATION_GUIDE.md

monGARS LLM Alignment Dataset Generator

The script scripts/generate_monGARS_llm_dataset.py creates a specialized dataset for fine-tuning by analyzing the project's own source code and documentation. This allows the LLM to learn about the monGARS architecture, API, and operational procedures.

The generation process is categorized to create diverse instruction-response pairs.

This diagram shows the data sources used by the generator to build the alignment dataset.

The script generates records across several categories:

Category	Description	Source
module_reference	Summarizes Python modules, including docstrings, classes, functions, and dependencies.	collect_module_summaries, build_module_examples
architecture	Creates Q&A pairs based on predefined architectural scenarios like conversation flows and memory consolidation.	ARCHITECTURE_SCENARIOS, build_architecture_examples
operations	Generates runbooks and triage steps from predefined operational playbooks (e.g., latency spikes, worker crashes).	OPERATIONS_PLAYBOOKS, build_operations_examples
alignment	Designs dataset slices and evaluation metrics based on alignment goals like RAG accuracy and safety.	ALIGNMENT_PLANS, build_alignment_examples
api	Documents API endpoints, payloads, and testing strategies based on openapi.json.	load_openapi_document, build_api_examples

The final output is split into monGARS_llm_train.jsonl and monGARS_llm_val.jsonl for direct use in training pipelines.

Sources: scripts/generate_monGARS_llm_dataset.py:1010-1020, datasets/monGARS_llm/README.md

General-Purpose Dataset Pipeline

The scripts/dataset_prep.py script provides a highly configurable and robust pipeline for aggregating, processing, and validating datasets from various sources, primarily Hugging Face. It is designed for large-scale data preparation and includes advanced features.

Key Features & Configuration

Feature	Configuration Class	Description
Pipeline Control	PipelineConfig	Manages output directories, language selection, checkpointing, and validation splits.
Memory Management	MemoryConfig	Configures memory limits, garbage collection thresholds, and caching strategies to handle large datasets.
Deduplication	DedupConfig	Enables exact and near-duplicate detection using MinHash/LSH.
Quality Filtering	QualityConfig	Sets rules for text length, word count, and quality scores.
PII Detection	PIIConfig	Configures PII detection with regex patterns and redaction strategies.

This script can load datasets, apply custom transformations, filter for quality and PII, perform deduplication, and save the cleaned output, making it a powerful tool for preparing data from external sources.

Sources: scripts/dataset_prep.py:117-313

Dataset Curation

After generation, datasets undergo a manual curation process to ensure high quality. The CURATION_GUIDE.md outlines a systematic workflow for this task.

This flowchart illustrates the manual curation steps for datasets produced by the monGARS Deep Scanner.

Curation Steps:

1. Review QA Report: Inspect report.md for dataset totals, warnings, and sample payloads to spot issues.
2. Validate Provenance: Use provenance.csv to cross-check a sample of records against the source files to verify correctness.
3. Promote SFT Records: Prioritize clear user intents and helpful assistant behaviors in sft_dataset.jsonl. Use _meta tags for filtering and analysis.
4. Curate Agent Handoffs: Ensure instructions in agent_handoff_dataset.jsonl have valid corresponding JSON outputs.
5. Prepare Embeddings Corpus: Deduplicate and clean embeddings_corpus.jsonl to remove boilerplate.
6. Track Adjustments: Document all manual curation steps in a changelog for future reproducibility.

Sources: CURATION_GUIDE.md

Dataset Processing and Consumption

Once datasets are generated and curated, they are processed and tokenized for consumption by training pipelines.

Normalization and Tokenization

The monGARS/mlops/dataset.py module contains helpers to prepare instruction datasets for supervised fine-tuning.

• _normalise_prompt_completion: This function standardizes heterogeneous dataset records into a consistent {"prompt": ..., "completion": ...} format by extracting fields from common keys like instruction, question, input, output, and response.
• _tokenize_pair: This function tokenizes the prompt-completion pairs. It applies a chat template (specifically the Dolphin template) and creates labels for training, where the prompt tokens are masked with -100 to be ignored during loss calculation.
• prepare_instruction_dataset: This function orchestrates the loading of a dataset from Hugging Face, applies the normalization and tokenization steps, and returns a torch formatted Dataset.

Sources: monGARS/mlops/dataset.py:44-148

End-to-End Training Workflow

The scripts/auto_unsloth_llm2vec.py script automates the entire workflow from dataset creation to fine-tuning.

This sequence diagram shows the automated process of scanning the repository, building a dataset, and launching the fine-tuning job.

This script first scans the repository for LLM callsites and module interactions, then combines this data with any provided seed datasets. It uses build_unsloth_llm2vec_dataset to create the final training and validation splits. Finally, it invokes run_unsloth_finetune to start the training process.

Sources: scripts/auto_unsloth_llm2vec.py:208-238

Dataset Catalog and Versioning

To manage the lifecycle of curated datasets, monGARS uses a versioning catalog.

Catalog Architecture

The models/datasets/catalog.py module defines the DatasetCatalog class, which maintains a JSON index (catalog.json) of available dataset versions.

This diagram shows the relationship between the DatasetCatalog and DatasetVersion classes.

• DatasetVersion: A data class representing a specific, versioned snapshot of a dataset. It includes metadata such as a run_id, creation timestamp, file paths, record count, and governance information.
• DatasetCatalog: Manages the registration of new DatasetVersion instances. It assigns an auto-incrementing version number and stores the metadata in catalog.json. The latest() method provides easy access to the most recent version.

This catalog is crucial for reproducibility and for allowing training pipelines to programmatically access the latest curated data. The Docker Compose deployment mounts the catalog and the datasets into the application containers, ensuring that training jobs always have access to the correct data.

Sources: models/datasets/catalog.py, datasets/monGARS_llm/README.md

Conclusion

The dataset management and curation system in monGARS is a sophisticated, multi-faceted component that is fundamental to the project's ability to self-improve. By systematically generating data from its own codebase, providing clear curation guidelines, and maintaining a versioned catalog, the project ensures a steady supply of high-quality, relevant data for fine-tuning. This closed-loop process, from code to data to model, is a key architectural pattern that drives the evolution of the monGARS AI assistant.