Test Replay Engine

Relevant source files

The following files were used as context for generating this wiki page:

• internal/runner/executor.go
• internal/runner/service.go
• internal/runner/test_loading.go
• internal/tui/test_executor.go
• internal/runner/environment.go
• internal/runner/comparison.go
• internal/runner/mock_server.go

The Test Replay Engine is the core component of the Tusk Drift CLI responsible for executing recorded traces against a service under test. It orchestrates the entire test run, from starting the user's service and a local mock server to replaying HTTP requests, comparing responses, and reporting deviations. The engine is designed to be UI-agnostic, with its logic contained primarily within the internal/runner package, and is consumed by an interactive TUI for tusk run commands.

The engine's primary goal is to provide fast, deterministic, and realistic API tests by simulating the exact conditions of a recorded trace. It achieves this by managing the service lifecycle, mocking all external I/O, and intelligently grouping tests that require specific environmental setups.

Architecture Overview

The replay engine is orchestrated by the Executor, which coordinates several key components to manage a test run. The ServiceManager handles the application under test, while the MockServer provides deterministic responses for outbound calls. Tests are loaded from trace files and grouped by environment before being executed.

The following diagram illustrates the main components of the Test Replay Engine and their interactions.

Core Components

Executor

The Executor is the central orchestrator of the test replay process. It manages the entire lifecycle of a test run, including setting up the environment, running tests concurrently, and tearing down resources.

Key Responsibilities:

• Starting and stopping the test environment, which includes the user's service and the mock server.
• Executing individual tests (RunSingleTest) by sending the recorded request to the service.
• Managing test concurrency based on configuration.
• Collecting and returning test results.
• Handling server crashes and orchestrating retries.

The main fields of the Executor struct are outlined below:

Field	Type	Description
Cfg	*config.Config	The loaded application configuration.
Service	*ServiceManager	Manages the lifecycle of the service under test.
MockServer	*MockServer	The server that mocks outbound dependencies.
concurrency	int	The number of tests to run in parallel.
OnTestCompleted	func(...)	A callback function invoked after each test completes.

Sources: internal/runner/executor.go:34-52

Service Manager

The ServiceManager is responsible for managing the lifecycle of the user's service. It uses commands defined in the .tusk/config.yaml to start, stop, and check the readiness of the service.

Key Methods:

• Start(): Executes the service.start.command and waits for the service to become ready.
• Stop(): Executes the service.stop.command to gracefully shut down the service.
• IsReady(): Checks if the service is ready, typically by using the configured readiness check.
• Restart(): Stops and then starts the service.

Sources: internal/runner/service.go:65-155

Mock Server

The MockServer is a critical component for ensuring deterministic test replays. It runs a local Unix socket or TCP server that intercepts outbound calls made by the service under test. The Tusk Drift SDK, when in REPLAY mode, redirects all instrumented calls (e.g., database queries, external HTTP requests) to this mock server. The mock server then serves the exact responses that were recorded in the trace file, ensuring the test runs in a completely isolated and predictable environment.

Sources: internal/runner/mock_server.go:27-33, internal/runner/executor.go:121-129

Test Loading

Tests are defined as runner.Test structs, which are loaded from .jsonl trace files. Each line in a trace file can represent a complete test case. The LoadTestsFromTraces function is responsible for reading these files, parsing the JSON, and creating a slice of Test objects.

The Test struct contains all the information needed to replay a single interaction:

Field	Type	Description
TraceID	string	A unique identifier for the trace.
Request	Request	The recorded HTTP request to be replayed.
Response	Response	The expected HTTP response.
Spans	[]Span	A list of recorded outbound calls (e.g., DB queries) made during the original request.

Sources: internal/runner/test_loading.go:20-33, 50-96

Execution Flow

The test replay process begins when tusk run is executed. The interactive TUI model (testExecutorModel) initiates the flow by calling startExecution, which then hands off control to the Executor.

Environment Grouping

A key feature of the engine is its ability to group tests by their required environment. Some traces may be recorded with specific environment variables (e.g., feature flags). These are captured as suite_setup spans in the trace file.

The GroupTestsByEnvironment function analyzes all tests to find these suite_setup spans and partitions the tests into EnvironmentGroups. Each group contains a set of tests and the specific environment variables required to run them. The Executor then processes these groups sequentially, setting the correct environment variables, running all tests for that group, and then tearing down the environment before moving to the next. This ensures that tests are run under the correct conditions.

This diagram shows how tests are partitioned into environment groups.

Test Execution Sequence

The overall sequence involves setting up an environment for a group, running its tests concurrently, and then tearing it down.

Single Test Replay (RunSingleTest)

The RunSingleTest function is the heart of the replay engine. It takes a single runner.Test object and performs the following steps:

1. Constructs an http.Request from the test.Request data.
2. Sends the request to the running service.
3. Reads the http.Response from the service.
4. Calls compareResponses to check for deviations between the actual response and the test.Response.
5. Populates and returns a TestResult struct.

Sources: internal/runner/executor.go:156-224

Server Crash and Retry Logic

The replay engine includes logic to handle cases where a test causes the service under test to crash. This is managed by the testExecutorModel in the TUI.

1. If RunSingleTest returns an error, the TUI checks the server's health.
2. If the server is down, the test that just ran (and any other tests that were active concurrently) are marked as "crashed".
3. These tests are queued for a retry.
4. The Executor attempts to restart the service.
5. After all other tests in the environment group are finished, a "retry phase" is initiated.
6. The crashed tests are run again, one by one (sequentially), to definitively identify the culprit.

This ensures that a single problematic test doesn't cause the entire test suite to fail and provides a more robust testing experience.

Sources: internal/tui/test_executor.go:946-1002

Result Comparison

After a test is replayed, the compareResponses function is used to check for deviations. This function, located in internal/runner/comparison.go, performs a deep comparison of the expected and actual HTTP responses.

It intelligently handles dynamic data by providing options to ignore certain fields or use custom comparators. For example, it can ignore differences in fields that are expected to change, such as timestamps or UUIDs. The comparison logic uses the go-cmp library.

A TestResult is generated, which includes a Passed status and a list of Deviations if any are found.

TestResult Field	Type	Description
TestID	string	The ID of the test that was run.
Passed	bool	true if no deviations were found.
CrashedServer	bool	true if the test caused the service to crash.
RetriedAfterCrash	bool	true if this was a retry attempt after a crash.
Duration	int64	The execution time in milliseconds.
Deviations	[]Deviation	A list of detected differences.

Sources: internal/runner/comparison.go:21-100, internal/runner/executor.go:226-241

Conclusion

The Test Replay Engine in tusk-drift-cli is a sophisticated system for providing deterministic and isolated API testing. By combining service lifecycle management, I/O mocking, and intelligent test grouping, it faithfully reproduces recorded interactions. Its architecture, centered around the Executor and its components, allows for robust and concurrent test execution, with features like server crash handling and environment-specific test runs making it a powerful tool for regression testing.