Managing Model Changes
Relevant source files
The following files were used as context for generating this wiki page:
Managing model changes within the work-validity-eval repository involves a structured pipeline to ensure that updates to Large Language Models (LLMs) used for classification, normalization, and screening do not compromise the integrity of the project's empirically derived taxonomy. While the taxonomy itself remains frozen, the automation scripts are designed to handle model swaps by backing up previous states and re-executing the evaluation pipeline to maintain consistency between model outputs and the established 18-operation taxonomy.
The system relies on specific environment variables to define which model is used for different stages of the pipeline, such as gpt-5.4-mini for operational normalization and knowledge screening. When a model change occurs, the project provides a dedicated utility to recode tasks and rebuild comparison artifacts, ensuring that the validity and coverage analysis remains accurate relative to the new model's capabilities.
Pipeline Architecture for Model Updates
The model update process follows a sequential flow that prioritizes data safety through backups before re-running the core analytical scripts. The primary mechanism for this is the rerun_after_chat_model_change.sh script, which orchestrates the transition.
The diagram above illustrates the automated sequence triggered when a chat model is updated to ensure downstream artifacts are synchronized.
Sources: scripts/rerun_after_chat_model_change.sh:1-26, README.md:163-175
Core Components and Configuration
Model selection is controlled via environment variables, allowing developers to target specific LLM versions for different processing stages.
| Stage | Environment Variable | Default Model | Purpose |
|---|---|---|---|
| Task Normalization | NORMALIZE_TASK_MODEL | gpt-5.4-mini | Strips domain-specific nouns from O*NET tasks. |
| Knowledge Screening | ROUND2_KSCREEN_MODEL | gpt-5.4-mini | Filters tasks for "knowledge work" suitability. |
| Pattern Proposal | PROPOSE_WORK_PATTERNS_MODEL | gpt-5.4-mini | Drafts work-pattern labels from task statements. |
Sources: scripts/normalize_task_operations.py:48, scripts/round2_knowledge_screen.py:46, scripts/propose_work_patterns.py:24
Execution Flow and Data Consistency
When a chat model is changed, the system must reconcile existing cached results with the new model's logic. This is particularly relevant for the normalization and screening stages, which use LLM-based decision-making.
Task Recoding and Normalization
The normalize_task_operations.py script transforms O*NET task statements into domain-stripped operation phrases. This ensures that the operation (e.g., "debug software") is prioritized over the domain (e.g., "robotics"). When models change, this script can be re-run to ensure the normalization logic aligns with the new model's linguistic parsing.
Knowledge Work Screening
The round-2 knowledge screen (round2_knowledge_screen.py) applies a task-level filter to determine if a task is primarily cognitive and judgment-bearing. A model change requires a re-evaluation of these decisions to ensure that the criteria for "knowledge work" are applied consistently across the dataset.
Key decision values in the screening process:
- Include: Primarily cognitive, produces an artifact or recommendation.
- Exclude: Direct physical, manual, or routine clerical execution.
- Uncertain: Ambiguous hybrids where physical and judgment weights are equal.
Sources: scripts/round2_knowledge_screen.py:32-40, scripts/round2_knowledge_screen.py:50-100
Impact on Artifact Generation
Model changes directly affect the derived analysis artifacts, specifically the coverage metrics and benchmark comparisons.
Rebuilding Comparison Artifacts
The build_comparison.py script generates coverage signals by mapping benchmarks to operations. It calculates the salience_weight of operations based on O*NET metrics and links them to scored units in the benchmark registry. Because this mapping depends on the deterministic coding of tasks into operations (which may have changed during recoding), this script must follow any model-driven recoding.
Calculated fields during this stage:
salience_weight: Derived fromimportance_mean,relevance_pct, andcore_weight.coverage_tier: Categorized as Broad (≥4 units), Moderate (2-3), Weak (1), or Missing (0).
Sources: scripts/build_comparison.py:44-64, scripts/build_comparison.py:82-95
Documentation and Freezing
Following a successful rerun after a model change, the exact model snapshot must be documented. The repository uses docs/freeze_log/ to store these records, ensuring that the results used in formal publications are traceable to a specific model version and configuration.
Sources: scripts/rerun_after_chat_model_change.sh:26, README.md:175
Conclusion
Managing model changes is a controlled process centered on the rerun_after_chat_model_change.sh workflow. By integrating automated backups, LLM-driven recoding, and a rigorous re-calculation of coverage metrics, the project ensures that its evaluation of AI agents remains robust and reproducible, even as underlying foundation models evolve. This lifecycle management is essential for maintaining the validity of the 18-operation taxonomy and the resulting audit of AI capability claims.