Cassette Maintenance

Relevant source files

The following files were used as context for generating this wiki page:

• tests/CASSETTE_MAINTENANCE.md
• tests/http_transport_recorder.py
• tests/openai_cassettes/o3_pro_basic_math.json
• tests/sanitize_cassettes.py
• tests/openai_cassettes/consensus_step1_gpt5_for.json

This project uses HTTP cassettes to record and replay API interactions for testing purposes. This allows tests to run quickly and reliably in CI environments without making actual API calls. The system is designed to handle standard request matching as well as a more flexible "semantic matching" for certain models, ensuring that tests are not brittle and do not require constant re-recording due to minor prompt changes.

The core implementation for recording and replaying is located in tests/http_transport_recorder.py. Cassettes are stored as JSON files in directories like tests/openai_cassettes/ and tests/gemini_cassettes/.

Sources: tests/CASSETTE_MAINTENANCE.md

How Cassette Matching Works

When a test makes an HTTP request, the system first checks if a corresponding cassette exists. If it does, the recorded response is returned (replay mode). If not, a real API call is made, and the interaction is saved to a new cassette file (record mode). The matching logic depends on the model being requested.

The following diagram illustrates the high-level decision flow for handling a request during testing.

Sources: tests/CASSETTE_MAINTENANCE.md, tests/http_transport_recorder.py

Matching Strategies

There are two primary strategies for matching a request to a recorded interaction.

Sources: tests/CASSETTE_MAINTENANCE.md, tests/http_transport_recorder.py:145-156

1. Standard Matching (Exact Hash)

For most models, requests are matched based on the HTTP method, path, and an exact hash of the request body. Any change to the request body, no matter how small, will result in a different hash and cause a mismatch.

Sources: tests/CASSETTE_MAINTENANCE.md

2. Semantic Matching (o3 Models)

To avoid brittle tests that break with frequent system prompt updates, o3 models (and other designated models) use semantic matching. This strategy ignores frequently changing parts of the request, such as system prompts and conversation memory instructions, and generates a signature based only on the core, stable content.

The logic to identify these requests is in the _is_o3_model_request function.

# Source: tests/http_transport_recorder.py:101-104
def _is_o3_model_request(self, content_dict: dict) -> bool:
    """Check if this is an o3 or other semantic-matching model request."""
    model = content_dict.get("model", "")
    return model.startswith("o3")

The fields for matching are extracted by _extract_semantic_fields.

Ignored Fields (o3 Models)	Description
System prompts	The instructional text that defines the model's role and behavior.
Conversation memory	Instructions related to maintaining context across turns.
Follow-up guidance	Text prompting the user for continuation.
Token limits & metadata	Other non-essential parameters that don't affect the core question.

Sources: tests/CASSETTE_MAINTENANCE.md, tests/http_transport_recorder.py:106-143

Cassette File Structure

Cassettes are JSON files that store an array of interactions. Each interaction object contains the request and response from a single API call. Sensitive information, such as API keys, is redacted during the recording process.

Below is a snippet from a sanitized cassette file. Note the {REDACTED} placeholder for the API key.

// Source: tests/gemini_cassettes/consensus/step2_gemini25_flash_against/mldev.json
{
  "replay_id": "consensus/step2_gemini25_flash_against/mldev",
  "interactions": [
    {
      "request": {
        "method": "post",
        "url": "{MLDEV_URL_PREFIX}/models/gemini-2.5-flash:generateContent",
        "headers": {
          "Content-Type": "application/json",
          "x-goog-api-key": "{REDACTED}",
          "user-agent": "google-genai-sdk/{VERSION_NUMBER} {LANGUAGE_LABEL}/{VERSION_NUMBER}",
          "x-goog-api-client": "google-genai-sdk/{VERSION_NUMBER} {LANGUAGE_LABEL}/{VERSION_NUMBER}"
        },
        "body_segments": [
          // ... request body contents
        ]
      },
      "response": {
        "status_code": 200,
        "headers": {
          // ... response headers
        },
        "body_segments": [
          // ... response body contents
        ]
      }
    }
  ]
}

Sources: tests/gemini_cassettes/consensus/step2_gemini25_flash_against/mldev.json, tests/openai_cassettes/o3_pro_basic_math.json, tests/openai_cassettes/consensus_step1_gpt5_for.json

Cassette Sanitization

To prevent secrets from being committed to the repository, a sanitization process runs automatically after recording. This process replaces sensitive values like API keys with placeholders. The main script for this is sanitize_cassettes.py, which looks for keys like authorization and x-goog-api-key and redacts their values.

The following diagram shows where sanitization fits into the recording workflow.

Sources: tests/sanitize_cassettes.py, tests/CASSETTE_MAINTENANCE.md

How to Re-Record a Cassette

You must re-record a cassette when the request changes in a way that affects its signature.

• For standard models: Any change to the request body.
• For o3 models: Changes to the user's question or core parameters like model or temperature.

The re-recording process is as follows:

Sources: tests/CASSETTE_MAINTENANCE.md

Troubleshooting

Error	Cause	Solution
"No matching interaction found"	The request body has changed, causing a signature mismatch. For o3 models, this means the core user question or parameters have changed.	Re-record the cassette by deleting the old file and running the test with a real API key.
"Cassette file not found"	The cassette has not been recorded yet or was deleted.	Run the test with a real API key to record a new cassette.
CI Fails but Local Tests Pass	The cassette was recorded with uncommitted local code changes.	Commit all code changes before re-recording cassettes, then commit the new cassettes.

Sources: tests/CASSETTE_MAINTENANCE.md

Best Practices

• Keep Test Questions Simple: Use stable, simple questions to avoid eliciting different responses from the API over time.
• Document Recording Conditions: Add comments to tests explaining when and under what conditions a cassette was recorded.
• Test Both Modes: Always verify that a newly recorded cassette works in replay mode (e.g., by using a dummy API key).
• Do Not Commit Secrets: The recording system sanitizes keys automatically, but always double-check that no other sensitive data has been inadvertently recorded.

Sources: tests/CASSETTE_MAINTENANCE.md