Response Comparison
Relevant source files
The following files were used as context for generating this wiki page:
The Response Comparison system is a core component of the Tusk Drift CLI, responsible for validating the behavior of a service under test. It compares the actual HTTP response received during a test replay with the expected response recorded in a trace file. The primary goal is to detect meaningful deviations in status codes, headers, and body content, while intelligently ignoring dynamic data such as UUIDs and timestamps. This ensures that tests are both robust and deterministic, failing only on genuine regressions. The comparison logic is primarily located in the internal/runner directory.
Sources: README.md, CONTRIBUTING.md, internal/runner/comparison.go
Core Comparison Flow
The main entry point for comparison is the Executor.compareAndGenerateResult method. This function orchestrates the entire comparison process for a single test. It takes the Test data (containing the expected response) and the *http.Response (the actual response from the service) and produces a TestResult object.
The following diagram illustrates the high-level flow of the comparison process.
The process involves parsing the actual response according to trace metadata, then sequentially comparing the status code, headers, and body. Any mismatches that are not ignored by the configuration are collected as Deviation objects, which determine if the test passed or failed.
Sources: internal/runner/executor.go:440-471, internal/runner/comparison.go:13-90
Body Content Parsing
A crucial step before comparison is correctly parsing the response body. The system uses a decodedType field from the core.Span protobuf schema (part of the trace data) to determine how to interpret the body bytes. This ensures the actual response is parsed in the same way the expected response was during trace recording. The parseDecodedBytes function handles this logic.
decodedType Value | Behavior |
|---|---|
DECODED_TYPE_JSON | The body is parsed as a JSON object or array. |
DECODED_TYPE_PLAIN_TEXT | The body is treated as a raw string. No JSON parsing is attempted, even if the content is valid JSON. |
DECODED_TYPE_UNSPECIFIED | The system first attempts to parse the body as JSON. If that fails, it falls back to treating it as a raw string. |
| (Not specified) | If no spans are available or the decodedType is not set, the behavior defaults to DECODED_TYPE_UNSPECIFIED. |
This mechanism is verified in comparison_test.go, which confirms that PLAIN_TEXT is not parsed as JSON and that UNSPECIFIED correctly falls back to string comparison for non-JSON data.
Sources: internal/runner/comparison.go:21-36, internal/runner/comparison_test.go:113-195
Recursive Body Comparison
The core of the body comparison is the compareJSONValues function, which recursively traverses and compares two data structures (typically map[string]any or []any).
This diagram illustrates the recursive logic of the comparison.
compareMaps: This function iterates through all keys in theexpectedmap, ensuring they exist in theactualmap and their values are equal (via a recursive call). It also checks for any extra keys in theactualmap that are not in theexpectedmap, which would cause a deviation unless ignored by the configuration.compareSlices: This function verifies that both slices have the same length and then compares each element in order, recursively callingcompareJSONValues.- Primitives: For primitive types (string, number, boolean), it first attempts a direct equality check using
safeEqual. If they are not equal, it then consults theDynamicFieldMatcherto see if the mismatch should be ignored.
Sources: internal/runner/comparison.go:138-243
Dynamic Field Matching
To handle data that naturally changes between runs (like IDs, timestamps, or request-specific tokens), the comparison logic uses a DynamicFieldMatcher. This component is responsible for deciding whether a mismatch in a primitive value should be ignored.
The matcher is configured via the comparison section of the .tusk/config.yaml file. The compareResponseBodies function loads this configuration and initializes the matcher.
| Configuration Key | Description |
|---|---|
ignore_uuids | (boolean) If true, ignores values that match the UUID format. |
ignore_timestamps | (boolean) If true, ignores values that match common timestamp formats (e.g., ISO 8601). |
ignore_dates | (boolean) If true, ignores values that match date formats. |
ignore_fields | (array of strings) A list of field names (case-insensitive) to ignore. This is useful for ignoring extra fields in the actual response that are not present in the expected one. |
ignore_patterns | (array of strings) A list of regex patterns. If a field's value matches a pattern, it is ignored. |
The ShouldIgnoreField method is the decision-maker, checking a field's name and value against the configured rules. For example, a test can pass if an id field has a different UUID in the actual response than the expected one, provided ignore_uuids is enabled.
Sources: internal/runner/comparison.go:92-136, internal/runner/comparison_test.go:207-230
Deviation Reporting
When a mismatch is found and not ignored, a Deviation is created. This struct captures the details of the failure, including the field path, expected value, and actual value.
The compareAndGenerateResult function aggregates deviations from three sources:
- Status Code: If
actualResp.StatusCode != test.Response.Status. - Headers: If an expected header is missing or has a different value.
- Body: If
compareResponseBodiesreturnsfalse, a single deviation for the entire body is created.
The final TestResult object contains a list of all Deviations and a Passed flag, which is true only if the list of deviations is empty. This result object is then used for reporting test outcomes to the user via the OutputSingleResult function.
Sources: internal/runner/comparison.go:40-80, internal/runner/executor.go:440-471, 594-650
Summary
The Response Comparison system in Tusk Drift CLI is a sophisticated mechanism for validating API test replays. It combines strict comparison of status codes and headers with a flexible, recursive, and configurable body comparison engine. By intelligently parsing response bodies based on trace metadata and allowing dynamic fields to be ignored, it provides a powerful tool for creating reliable, low-maintenance regression tests from real-world traffic.