Test Strategy and Fixtures

Relevant source files

The following files were used as context for generating this wiki page:

• AGENTS.md
• tests/README.md
• tests/unit/transports/web_legacy/test_getscores_fixtures.py
• tests/integration/test_getscores_status_fixtures.py
• tests/unit/athena_cli/stable_verification/test_replay_download.py
• src/athena_cli/stable_verification/score_submit.py
• tests/unit/transports/test_health.py
• tests/integration/test_sqlalchemy_score_repository.py

Athena employs a rigorous testing strategy designed to ensure compatibility with legacy osu! stable client protocols while maintaining high internal code quality. The strategy combines unit tests, integration tests, and a "Golden Fixture" verification system to validate externally observable behaviors against captured official client traffic.

The project enforces strict type safety (basedpyright strict mode) and linting rules (ruff) across all test suites, treating test code with the same level of discipline as production code. This includes the use of in-memory implementations and typed stubs instead of untyped mocks to prevent type erosion during verification.

Sources: AGENTS.md:46-49, tests/README.md:3-6

Testing Hierarchy and Methodology

Athena's testing architecture is divided into three primary tiers: unit tests for isolated logic, integration tests for persistence and service boundaries, and compatibility verification for protocol adherence.

Core Testing Tiers

Tier	Focus	Key Tools
Unit Tests	Isolated component logic, protocol parsing, and formatters.	pytest, InMemory repositories
Integration Tests	Persistence (SQLAlchemy/Postgres), service workflows, and E2E routes.	TestClient, real PostgreSQL
Compatibility	Verification of "Golden Fixtures" against official client behavior.	athena_cli.stable_verification

Sources: AGENTS.md:49, tests/integration/test_sqlalchemy_score_repository.py:1-6, tests/unit/transports/web_legacy/test_getscores_fixtures.py:1-12

Quality Gate Workflow

The local quality gate is managed via a centralized script to ensure consistency across development environments.

Stable Compatibility Verification

A significant portion of Athena's testing is dedicated to "Stable Verification." This involves comparing server responses against "Golden Fixtures"—raw or metadata-only captures of official Bancho responses.

Getscores Fixture Validation

Getscores verification ensures that the server correctly maps internal beatmap statuses to the specific integers required by legacy clients.

• Status Mapping: Ranked (2), Loved (5), Qualified (4), and Pending/WIP/Graveyard (0).
• Structure: Must have exactly 4 data lines (status, offset, display, rating) followed by empty placeholders.
• Official Precedence: Athena follows official behavior over other private server implementations (e.g., returning full headers for Pending maps instead of short "0|false" responses).

Sources: tests/unit/transports/web_legacy/test_getscores_fixtures.py:101-140, tests/integration/test_getscores_status_fixtures.py:1-15

Replay Download and Score Submission

Verification for more complex flows like replay downloads involves diagnostic pipelines to classify integrity.

Fixture Management and Architecture

Athena uses a sophisticated directory structure and JSON schema to manage fixtures, ensuring they do not leak sensitive information (secrets) while remaining useful for verification.

Fixture Structure

Fixtures are located in tests/fixtures/stable_compatibility/ and are organized by surface area.

• Metadata Records: Store observed headers, statuses, and byte sizes without raw body payloads when possible.
• Secret Policy: Projects often use metadata-only policies to avoid committing encrypted payloads or session tokens.
• Diagnostic Classification: Replay blobs are diagnosed as INTEGRITY_PASS, MISSING_SCORE, MISSING_REPLAY, or STORAGE_INTEGRITY_FAILURE.

Sources: tests/unit/athena_cli/stable_verification/test_replay_download.py:13-30, src/athena_cli/stable_verification/score_submit.py:133-150

Technical Implementation: Fake Connection Pattern

For infrastructure unit tests (like health checks), Athena utilizes a "Fake" pattern to simulate database or cache failures without high-overhead mocks.

@final
class FakeValkey:
    def __init__(self, should_fail: bool) -> None:
        self._should_fail = should_fail

    async def ping(self) -> str:
        if self._should_fail:
            raise ConnectionError("valkey down")
        return "PONG"

Sources: tests/unit/transports/test_health.py:72-80

Type Safety in Tests

The project mandates specific practices to maintain type safety within the test suite, discouraging the use of AsyncMock.

1. InMemory Implementations: Prefer InMemoryUserRepository or InMemorySessionStore.
2. Typed Fakes: Create classes compliant with specific Protocols (e.g., HIBP clients).
3. Typed Factories: Use centralized factory functions in tests/factories/ (e.g., make_user()) instead of raw dictionaries.
4. Runtime Assertions: Use specialized helpers for testing frozen objects or forbidden operations.

Sources: tests/README.md:12-45

Summary

Athena's test strategy is anchored by the dual requirement of high code quality and absolute protocol compatibility. By combining strict type-safe unit testing with a comprehensive verification system for legacy fixtures, the project ensures that internal refactoring does not break externally observable client behavior. The use of "Golden Fixtures" and diagnostic classifications allows the team to track compatibility progress and gaps with technical precision.