Fitness Evaluation & Benchmarking

Relevant source files

The following files were used as context for generating this wiki page: - [packages/core/src/evaluation/calculate-fitness/calculateFitness.ts](https://github.com/eenlars/lucky/blob/HEAD/packages/core/src/evaluation/calculate-fitness/calculateFitness.ts) - [packages/core/src/evaluation/benchmarks/swe/test_evaluation_input.ts](https://github.com/eenlars/lucky/blob/HEAD/packages/core/src/evaluation/benchmarks/swe/test_evaluation_input.ts) - [packages/core/src/improvement/gp/__tests__/Evaluator.test.ts](https://github.com/eenlars/lucky/blob/HEAD/packages/core/src/improvement/gp/__tests__/Evaluator.test.ts) - [apps/web/src/lib/migrations.ts](https://github.com/eenlars/lucky/blob/HEAD/apps/web/src/lib/migrations.ts) - [packages/core/src/prompts/improveWorkflow.p.ts](https://github.com/eenlars/lucky/blob/HEAD/packages/core/src/prompts/improveWorkflow.p.ts) - [packages/core/src/prompts/workflowAnalysisPrompts.ts](https://github.com/eenlars/lucky/blob/HEAD/packages/core/src/prompts/workflowAnalysisPrompts.ts) - [packages/core/src/evaluation/calculate-fitness/__tests__/calculateFitness.test.ts](https://github.com/eenlars/lucky/blob/HEAD/packages/core/src/evaluation/calculate-fitness/__tests__/calculateFitness.test.ts) - [apps/web/src/app/components/evolution/WorkflowEvolutionVisualization.tsx](https://github.com/eenlars/lucky/blob/HEAD/apps/web/src/app/components/evolution/WorkflowEvolutionVisualization.tsx)

The Fitness Evaluation and Benchmarking system is a core component of the lucky project's evolutionary capabilities. Its primary purpose is to quantitatively measure the performance of AI agent workflows. By assigning a "fitness" score to each workflow variation, the system provides the necessary feedback for both iterative (Cultural) and population-based (Genetic Programming) evolution to select and breed better-performing workflows.

This system evaluates workflows against multiple dimensions, including accuracy, execution cost, and time. It also includes a framework for benchmarking against standardized datasets like SWE-bench to provide objective performance metrics. The resulting fitness scores and feedback are used by various improvement modules to guide the optimization process.

Fitness Calculation

The central function for scoring a workflow's performance is calculateFitness. This function takes the output of a workflow execution and returns a multi-faceted fitness score.

Sources: packages/core/src/evaluation/calculate-fitness/calculateFitness.ts

Process Flow

The fitness calculation process involves an AI-driven accuracy assessment followed by a weighted calculation of the final score.

The diagram below shows the data flow within the calculateFitness function.

Key Steps & Logic

1. Input Validation: The function first checks if agentSteps and finalWorkflowOutput are provided. If not, it returns a score of 0, gracefully handling executions with no output.
   Sources: packages/core/src/evaluation/calculate-fitness/calculateFitness.ts:29-39

2. AI-based Accuracy Assessment: The core of the accuracy measurement is an AI call. The workflow's output, ground truth (evaluation), and an optional rubric are sent to an AI model. The model is prompted to return a structured response containing an accuracy score (0-100) and feedback.
   Sources: packages/core/src/evaluation/calculate-fitness/calculateFitness.ts:48-68

3. Score Calculation: The final fitness score is a weighted sum of three components:

   • Accuracy Score: The accuracy returned by the AI.
   • Time Bonus: A normalized score based on totalTime.
   • Cost Bonus: A normalized score based on the AI evaluation cost (response.usdCost).

   Crucially, the time and cost bonuses are "gated" by the accuracy. This means a workflow with low accuracy receives little to no bonus for being fast or cheap, preventing the evolution of incorrect but efficient workflows.

   // packages/core/src/evaluation/calculate-fitness/calculateFitness.ts:79-85
   const accuracyGate = Math.max(0, accuracy / 100)
   const gatedTimeBonus = normalizedTime * accuracyGate
   const gatedCostBonus = normalizedCost * accuracyGate
   
   const finalFitness =
     effectiveScore * config.improvement.fitness.weights.score +
     gatedTimeBonus * config.improvement.fitness.weights.time +
     gatedCostBonus * config.improvement.fitness.weights.cost
   

   The final score is capped at 100.
   Sources: packages/core/src/evaluation/calculate-fitness/calculateFitness.ts:78-88

Data Structures

The process relies on specific data structures for input and output.

FitnessFunctionInput

This interface defines the inputs required to calculate fitness.

Parameter	Type	Description
agentSteps	AgentStep[]	The sequence of steps taken by the agent.
totalTime	number	Total execution time in milliseconds.
totalCost	number	Total execution cost in USD.
evaluation	string	The ground truth or expected output for comparison.
outputSchema	object	Optional JSON schema for the expected output format.
finalWorkflowOutput	string	The final output produced by the workflow.
overrideModel	string	Optional model to override the default fitness model.
rubric	string	Optional rubric for the AI to use during evaluation.
Sources: packages/core/src/evaluation/calculate-fitness/fitness.types.ts

FitnessOfWorkflow

This is the structured output of the fitness calculation.

Field	Type	Description
score	number	The final weighted fitness score (0-100).
totalCostUsd	number	The total cost, including the fitness evaluation AI call.
totalTimeSeconds	number	The total time in seconds.
accuracy	number	The accuracy score (0-100) from the AI evaluator.
Sources: packages/core/src/evaluation/calculate-fitness/fitness.types.ts, packages/core/src/evaluation/calculate-fitness/calculateFitness.ts:90-96

The AI response is validated against FitnessOfWorkflowSchema, a Zod schema that ensures the AI provides the required accuracy field.
Sources: packages/core/src/evaluation/calculate-fitness/calculateFitness.ts:65

Benchmarking

The system includes a framework for benchmarking workflows against standardized software engineering tasks, most notably SWE-bench.

Sources: packages/core/src/evaluation/benchmarks/swe/test_evaluation_input.ts

SWE-Bench Evaluation

The test_evaluation_input.ts script outlines a complete end-to-end process for evaluating a workflow on a SWE-bench task.

SWE-Bench Process Flow

This diagram illustrates the steps involved in running an evaluation against a SWE-bench instance.

Key Components

1. Dataset Loading: The script can download the SWE-bench dataset from Hugging Face if a HUGGING_FACE_API_KEY is present. It looks for .jsonl, .json, or .parquet files.
   Sources: packages/core/src/evaluation/benchmarks/swe/test_evaluation_input.ts:44-106

2. AI Solver (solveWithAI): This function sends the problem statement from the benchmark to an AI model, instructing it to provide a structured response containing an analysis, a code solution/patch, reasoning, and testing considerations.
   Sources: packages/core/src/evaluation/benchmarks/swe/test_evaluation_input.ts:222-299

3. AI Validator (validateAISolution): After the AI generates a solution, this function sends both the AI's solution and the ground truth solution (from the benchmark data) to another AI model acting as an expert code reviewer. This "judge" AI evaluates the solution based on several criteria.
   Sources: packages/core/src/evaluation/benchmarks/swe/test_evaluation_input.ts:108-220

Validation Criteria

The AI-generated solution is scored against the ground truth using the following criteria:

Criterion	Description
Has Code Changes	Does the AI solution contain actual code changes/patches?
Addresses Problem	Does the AI solution directly address the problem statement?
Format Quality	Is the solution well-formatted and clear?
Test Awareness	Does the AI solution mention or consider relevant tests?
Ground Truth Similarity	How similar is the AI approach to the ground truth solution? (Calculated using Jaccard similarity).
Meets Evaluation Goal	Does the solution meet the specific evaluation goal provided?
AI Confidence	The confidence score (0-10) provided by the solving AI.
Sources: packages/core/src/evaluation/benchmarks/swe/test_evaluation_input.ts:153-160, 185-199

Integration with Evolution

Fitness data is a critical input for the evolutionary algorithms that improve workflows.

Genetic Programming (GP) Evaluator

In the GP engine, the GPEvaluatorAdapter is responsible for evaluating a Genome (a workflow variant). It wraps an AggregatedEvaluator, which in turn likely calls calculateFitness.

The following sequence diagram shows the interactions during a genome evaluation.

After evaluation, the fitness score and feedback are stored on the Genome instance, allowing the selection mechanism of the GP engine to identify and prioritize high-performing individuals for reproduction.
Sources: packages/core/src/improvement/gp/tests/Genome.test.ts:370-381

Cultural Evolution Prompts

In cultural evolution (iterative improvement), the fitness metrics are directly embedded into prompts that ask an AI to act as a "judge" or "mechanic" to suggest improvements.

FITNESS METRICS:
- Score: ${fitness.score}/100
- Total Cost: $${fitness.totalCostUsd.toFixed(2)}
- Total Time: ${fitness.totalTimeSeconds}s
- Data Accuracy: ${fitness.accuracy}

This context helps the AI understand the workflow's current performance and identify bottlenecks.
Sources: packages/core/src/prompts/improveWorkflow.p.ts:36-44, packages/core/src/prompts/workflowAnalysisPrompts.ts:88-93

Fitness Data Migration

The structure of the stored fitness data has evolved over time. The apps/web/src/lib/migrations.ts file contains logic to handle and normalize these different formats.

This diagram shows the logic for extracting accuracy and fitness from various historical data formats.

The migration script migrateFitnessToColumns reads WorkflowInvocation rows from the database, uses extractFitnessValues to parse the JSON fitness blob, and updates the dedicated accuracy and fitness columns in the table. This normalizes the data for easier querying and visualization.
Sources: apps/web/src/lib/migrations.ts:97-154

The evolution visualization components also contain logic to handle these different data shapes, ensuring that historical runs can still be displayed correctly.
Sources: apps/web/src/app/components/evolution/WorkflowEvolutionVisualization.tsx:75-99

Summary

The Fitness Evaluation and Benchmarking system is the sensory organ of the lucky project's evolutionary core. It provides the objective, multi-dimensional feedback necessary for the system to learn and improve. Through a combination of AI-driven accuracy scoring, weighted calculations for efficiency, and a robust benchmarking framework, it effectively guides the search for more accurate, cheaper, and faster AI workflows. Its integration into both genetic and cultural evolution mechanisms makes it a cornerstone of the platform's autonomous optimization capabilities.