Testing Strategy

Relevant source files

The following files were used as context for generating this wiki page: - [apps/web/src/__tests__/shared/README.md](https://github.com/eenlars/lucky/blob/HEAD/apps/web/src/__tests__/shared/README.md) - [tests/README.md](https://github.com/eenlars/lucky/blob/HEAD/tests/README.md) - [packages/core/src/__tests__/TESTING.md](https://github.com/eenlars/lucky/blob/HEAD/packages/core/src/__tests__/TESTING.md) - [packages/core/src/messages/api/__tests__/shared/testHelpers.ts](https://github.com/eenlars/lucky/blob/HEAD/packages/core/src/messages/api/__tests__/shared/testHelpers.ts) - [apps/examples/definitions/human-approval/__tests__/tool.test.ts](https://github.com/eenlars/lucky/blob/HEAD/apps/examples/definitions/human-approval/__tests__/tool.test.ts) - [packages/core/src/utils/validation/README.md](https://github.com/eenlars/lucky/blob/HEAD/packages/core/src/utils/validation/README.md) - [packages/core/src/messages/pipeline/__tests__/InvocationPipeline.spec.test.ts](https://github.com/eenlars/lucky/blob/HEAD/packages/core/src/messages/pipeline/__tests__/InvocationPipeline.spec.test.ts) - [apps/web/src/app/api/dev/pipeline/invoke/route.ts](https://github.com/eenlars/lucky/blob/HEAD/apps/web/src/app/api/dev/pipeline/invoke/route.ts) - [packages/tools/PROBLEM_STATEMENT.md](https://github.com/eenlars/lucky/blob/HEAD/packages/tools/PROBLEM_STATEMENT.md) - [packages/core/src/workflow/setup/__tests__/resultPersistence.test.ts](https://github.com/eenlars/lucky/blob/HEAD/packages/core/src/workflow/setup/__tests__/resultPersistence.test.ts) - [README.md](https://github.com/eenlars/lucky/blob/HEAD/README.md)

The project employs a multi-layered testing strategy to ensure the reliability, correctness, and performance of its autonomous agentic workflows. The strategy encompasses unit, integration, and end-to-end (E2E) tests, managed primarily with the Vitest testing framework. A significant emphasis is placed on creating a smooth developer experience through shared test infrastructure, comprehensive mocking, and clear patterns for testing different system components like tools, workflows, and core logic.

The testing pipeline is deeply integrated into the development lifecycle with pre-commit and pre-push hooks to maintain code quality. This ensures that critical paths are validated before code is integrated, supporting the project's goal of building a robust evolutionary optimization framework for multi-agent systems.

Sources: tests/README.md, apps/web/src/tests/shared/README.md, README.md:19-22

Test Types

The repository organizes tests into three main categories, each serving a distinct purpose in the validation process.

Sources: tests/README.md:23-25

This diagram illustrates the hierarchical structure of the testing strategy.

Test Type	Location	Purpose
Unit Tests	Co-located with source (packages/{package}/src/)	Test individual functions, classes, and pure logic in isolation.
Integration Tests	tests/integration/	Test how multiple modules work together with external dependencies like APIs and databases.
E2E Tests	tests/e2e-essential/	Test complete workflows and system behavior from end to end.

Sources: tests/README.md:5-12, tests/README.md:62-66

Unit Tests

Unit tests are co-located with the source code they are testing. They focus on small, isolated pieces of functionality. A key area for unit testing is the tool development framework, where each tool is tested in isolation.

An example is the test for the humanApproval tool, which verifies its configuration and parameter validation.

// Source: apps/examples/definitions/human-approval/__tests__/tool.test.ts:19-21
it("should have correct tool configuration", () => {
  expect(humanApproval.name).toBe("humanApproval")
  expect(humanApproval.execute).toBeDefined()
})

Another example is testing pure logic, such as the utility functions for Genetic Programming (GP).

// Source: packages/core/src/improvement/gp/__tests__/utils.test.ts:5-12
describe("Math and Statistics", () => {
  it("should calculate euclidean distance", () => {
    const _point1 = [1, 2, 3]
    const _point2 = [4, 5, 6]

    // euclidean distance formula: sqrt((x2-x1)² + (y2-y1)² + (z2-z1)²)
    const expectedDistance = Math.sqrt((4 - 1) ** 2 + (5 - 2) ** 2 + (6 - 3) ** 2)

    expect(expectedDistance).toBeCloseTo(5.196, 3)
  })
})

Sources: packages/tools/PROBLEM_STATEMENT.md:41-43, apps/examples/definitions/human-approval/tests/tool.test.ts, packages/core/src/improvement/gp/tests/utils.test.ts

Integration Tests

Integration tests are located in the tests/integration/ directory and are designed to verify that multiple modules interact correctly with external dependencies like APIs and databases. These tests cover the full HTTP request/response cycle for API routes and test the persistence layer.

Some integration tests, particularly those for the InvocationPipeline, make real calls to Large Language Models (LLMs) and are noted to be long-running and expensive. These tests validate that the system can correctly interpret prompts and execute tools in the specified order.

// Source: packages/core/src/messages/pipeline/__tests__/InvocationPipeline.spec.test.ts:101-109
// Execute the REAL pipeline - this will make actual LLM calls
await pipeline.prepare()
await pipeline.execute()
const result = await pipeline.process()

// Verify the pipeline succeeded
expect(result).toBeDefined()
if (result.error) {
  throw new Error(`Pipeline execution failed: ${result.error.message}`)
}

Sources: tests/README.md:27-33, packages/core/src/messages/pipeline/tests/InvocationPipeline.spec.test.ts:11-13, 101-109

End-to-End (E2E) Tests

E2E tests, found in tests/e2e-essential/, validate complete user workflows. They are divided into two categories:

• Smoke Tests (smoke/): Fast-running tests that cover critical path validation. These are executed on every commit.
• Gate Tests (gate/): More comprehensive tests that run before any code is pushed to the main branch.

A key part of the E2E strategy is the use of "Golden Traces," which are stored execution traces used for regression testing to ensure workflow execution remains consistent.

Sources: tests/README.md:40-45, README.md:112-114

Test Infrastructure and Helpers

The project has a robust shared test infrastructure to reduce code duplication and improve maintainability. This is primarily documented in apps/web/src/__tests__/shared/README.md and packages/core/src/__tests__/TESTING.md.

Shared Test Utilities

A centralized set of test utilities, mocks, and helpers is located in apps/web/src/__tests__/shared/.

This diagram shows the directory structure of the shared test infrastructure.

Key components of this infrastructure include:

Component	Description
setupGPTestMocks()	Provides a complete setup for genetic programming tests, replacing ~50 lines of boilerplate.
mockRuntimeConstants()	A base mock function with deep merge support for CONFIG, PATHS, and MODELS.
mockRuntimeConstantsForGP()	An optimized mock for GP tests with common settings.
processAndValidateSteps()	A helper to process and validate test results with common assertions.
createTestMessages()	A factory for creating standard test messages, reducing duplication of large context blocks.

Sources: apps/web/src/tests/shared/README.md:10-12, 21-28, 62-65, packages/core/src/messages/api/tests/shared/testHelpers.ts:13-19, 61-64

Mocking Strategy

The testing strategy relies heavily on mocking external dependencies to isolate the code under test. The packages/core/src/__tests__/TESTING.md guide provides templates for mocking various components.

The general approach is to use vi.mock to replace modules. A common pattern is to use a beforeEach block to set up mocks for each test.

// Source: packages/core/src/__tests__/TESTING.md:5-15
import { describe, it, expect, beforeEach, vi } from "vitest"
import {
  setupCoreTest,
  mockRuntimeConstantsForGP,
  // ...
} from "@core/utils/__tests__/setup/coreMocks"

describe("MyFunction", () => {
  beforeEach(() => {
    setupCoreTest() // Always include this
    // Add mockRuntimeConstantsForGP() if your function uses CONFIG.*
  })
})

The following diagram shows a typical sequence for setting up a test with shared mocks.

Common Test Failures and Fixes

The testing documentation provides guidance on resolving common issues.

Error	Fix
Cannot read property X of undefined	Add X to the CONFIG mock or use mockRuntimeConstantsForGP().
Module not found / undefined is not a function	Check that import paths match tsconfig.json aliases.
Expected spy to be called	Use vi.mocked(Module) to access mock functions.
Global mock conflicts	Create isolated tests with minimal mocking instead of fighting global mocks.

Sources: packages/core/src/tests/TESTING.md:102-108, 140-145

Workflow and Pipeline Testing

Given the project's focus on agentic workflows, a significant portion of the testing effort is dedicated to validating workflow configurations and the InvocationPipeline.

Workflow Validation

Before execution, workflow configurations undergo a multi-stage validation process. This acts as an automated test suite for the workflow's structural integrity.

This diagram shows the steps in the workflow validation pipeline.

Pipeline Invocation Testing

A dedicated API endpoint, POST /api/dev/pipeline/invoke, exists for development-time testing of the InvocationPipeline. This allows developers to send a test request with a system prompt, message, and tool configuration to execute a pipeline and receive the results.

The following sequence diagram illustrates the flow of a request to this test endpoint.

File Persistence Testing

Unit tests also cover core functionalities like persistence. For example, resultPersistence.test.ts verifies that workflow configurations are correctly saved to the file system, including the creation of timestamped backups. It uses helpers to manage the test file system state, ensuring tests are isolated.

// Source: packages/core/src/workflow/setup/__tests__/resultPersistence.test.ts:133-148
it("should create a timestamped backup file", async () => {
  await persistWorkflow(testWorkflowConfig, testFileName)

  // check that backup directory has files
  const backupFiles = await fs.readdir(actualBackupDir)
  const backupName = path.basename(testFileName, path.extname(testFileName))
  const testBackups = backupFiles.filter(f => f.startsWith(`${backupName}_`)).sort() // filenames contain ISO timestamps; lexicographic sort orders by time

  expect(testBackups.length).toBeGreaterThan(0)

  // verify backup content (check the most recent backup for this test)
  const latestBackup = testBackups[testBackups.length - 1]
  const backupPath = path.join(actualBackupDir, latestBackup)
  const content = await fs.readFile(backupPath, "utf-8")
  const parsed = JSON.parse(content)
  expect(parsed).toEqual(testWorkflowConfig)
})

Sources: packages/core/src/workflow/setup/tests/resultPersistence.test.ts

CI/CD Integration

The testing strategy is enforced through CI/CD automation using Husky hooks and GitHub Actions.

• Pre-commit: Runs fast smoke tests (bun run test:smoke).
• Pre-push: Runs gate tests (bun run test:gate) and type checking (bun run tsc).

The GitHub Actions workflow for testing runs on every push and pull request, executing the full test suite with coverage reporting.

# Source: apps/web/src/__tests__/shared/README.md:208-225
# .github/workflows/test.yml
name: Test Suite

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: oven-sh/setup-bun@v1

      - name: Install dependencies
        run: bun install

      - name: Run tests
        run: bun run test --coverage

Sources: tests/README.md:96-102, apps/web/src/tests/shared/README.md:208-225

Conclusion

The testing strategy for lucky is comprehensive, covering multiple layers of validation from individual units to complete end-to-end workflows. It is built around a robust infrastructure of shared helpers and mocks that streamline the development process. By integrating tests directly into the CI/CD pipeline, the project ensures a high standard of quality and reliability, which is crucial for a system designed for autonomous and evolutionary computation.