Tool Experiment Framework
Relevant source files
The following files were used as context for generating this wiki page:
The Tool Experiment Framework is a data-driven system within the Engineering Platform designed for A/B testing development tools, configurations, and workflows. It enables engineering leadership to make evidence-based decisions regarding the standardization or deprecation of tools based on objective metrics such as code quality, velocity, and developer satisfaction. Sources: README.md:14, experiments/tool-experiments.json:3
The framework operates by assigning engineers or projects into treatment and control cohorts to compare experimental tools against current standards. This systematic approach ensures that the "Standard" toolset remains optimal and evolves according to verified performance improvements rather than subjective preference. Sources: experiments/experiment-schema.json:71-89, tools-registry.json:448-456
Architecture and Data Flow
The framework relies on a centralized registry and structured experiment definitions to track progress and calculate statistical significance.
Component Interaction
The lifecycle of an experiment involves several interconnected files and scripts:
- Tools Registry: Defines all available tools and their current status (Standard, Experimental, Deprecated). Sources: tools-registry.json:7-14
- Experiment Definitions: JSON files following a strict schema that define cohorts, hypotheses, and success criteria. Sources: experiments/experiment-schema.json:7-122
- Management Scripts: CLI tools for transitioning experiment states and recording checkpoints. Sources: scripts/manage-experiments.js:150-205
- Analytics Engine: An MCP server component that processes raw git and profile data to generate effectiveness reports. Sources: mcp-servers/team-analytics/tools/tool-effectiveness-tools.js:107-160
This diagram illustrates the flow from tool registration through experiment execution to final promotion.
Data Models and Schemas
Experiment Schema
The experiment schema enforces strict validation for all A/B tests. Every experiment must define a falsifiable hypothesis and specific success thresholds.
| Field | Type | Description |
|---|---|---|
id | String | Unique ID following pattern exp-001-name. |
status | Enum | State: planned, active, paused, completed, abandoned. |
cohorts | Object | Defines treatment (new tool) and control (baseline). |
metrics_to_track | Array | List of metrics (e.g., code_quality_score, bug_rate). |
success_criteria | Object | Thresholds for min_improvement and max_p_value. |
Sources: experiments/experiment-schema.json:16-122
Tools Registry Statuses
Tools within the framework are categorized by their lifecycle status:
| Status | Description |
|---|---|
| Standard | Recommended for all projects; proven effective. |
| Experimental | Available for testing; data collection phase. |
| Deprecated | Being phased out; migration required. |
| Proposed | Under evaluation; not yet available for testing. |
Sources: tools-registry.json:415-420
Experiment Management Lifecycle
The manage-experiments.js script provides a CLI for managing the lifecycle of an experiment. The process transitions through four main stages:
1. Initialization (start)
Sets the start and end dates based on the defined duration_weeks and transitions the experiment status to active. It initializes the cohorts based on assigned engineers. Sources: scripts/manage-experiments.js:41-63
2. Monitoring (checkpoint)
Weekly snapshots are recorded to track progress. This process involves querying the team-analytics MCP for metrics related to code quality, velocity, and PR approval rates for each cohort. Sources: scripts/manage-experiments.js:66-107
3. Completion and Analysis (complete)
Once the duration is met, the experiment is moved to completed_experiments. A result object is generated containing the conclusion (treatment_wins, control_wins, no_difference, or inconclusive). Sources: scripts/manage-experiments.js:110-148, experiments/experiment-schema.json:173-180
4. Reporting (report)
Generates a comprehensive summary including delta calculations ($\Delta$) between treatment and control metrics. Sources: scripts/manage-experiments.js:150-205
Metrics and Evaluation
The framework tracks various technical and qualitative metrics to determine tool effectiveness.
Tracked Metrics
- Technical:
code_quality_score,velocity_commits_per_week,bug_rate,test_coverage_percent,pr_approval_rate. - DORA:
deployment_frequency,lead_time_hours. - Qualitative:
developer_satisfaction(1-5 scale).
Sources: experiments/experiment-schema.json:71-89, tools-registry.json:423-445
Promotion Criteria
An experimental tool is recommended for promotion to "Standard" if it meets specific thresholds defined in the registry:
- Minimum Users: 5 engineers.
- Minimum Duration: 8 weeks.
- Improvement: $\ge 10%$ improvement in primary metrics.
- Statistical Significance: $p\text{-value} < 0.05$.
- Satisfaction: $\ge 4.0$ score.
Sources: tools-registry.json:448-456, mcp-servers/team-analytics/tools/tool-effectiveness-tools.js:296-320
Tool Effectiveness Analysis
The analyze-tool-effectiveness.js script and the ToolEffectivenessEngine class within the team-analytics MCP provide the logic for comparing tools.
Core Functions
getToolEffectiveness(category): Ranks tools within a category by a primary metric and determines a winner against the baseline. Sources: mcp-servers/team-analytics/tools/tool-effectiveness-tools.js:107-160calculateImprovement(before, after): Calculates the percentage delta for specific metrics following a tool change. Sources: mcp-servers/team-analytics/tools/tool-effectiveness-tools.js:229-242recommendToolPromotion(toolId): Checks a specific tool against the promotion criteria requirements. Sources: mcp-servers/team-analytics/tools/tool-effectiveness-tools.js:264-306
Analysis Report Example
The analysis script generates a comparison table showing how experimental tools perform against the current standard.
// Example analysis logic for a winner vs baseline
const improvement = ((winner.code_quality - baseline.code_quality) / baseline.code_quality * 100).toFixed(1);
if (improvement >= 10 && winner.user_count >= 3) {
return `✅ Recommendation: PROMOTE ${winner.name} to standard`;
}
Sources: scripts/analyze-tool-effectiveness.js:115-125
Conclusion
The Tool Experiment Framework provides a rigorous, data-centric approach to engineering productivity. By quantifying the impact of tools like cursor vs vscode or eslint-strict vs eslint-base, the platform ensures that the engineering team's toolkit is continuously optimized for quality and speed while minimizing the risks associated with unvalidated migrations. Sources: README.md:14-16, tools-registry.json:448-461