Stable Compatibility Verifier

Relevant source files

The following files were used as context for generating this wiki page:

• src/athena_cli/stable_verification/__init__.py
• src/athena_cli/stable_verification/score_submit.py
• tests/unit/athena_cli/stable_verification/test_replay_download.py
• tests/unit/transports/web_legacy/test_getscores_fixtures.py
• tests/unit/athena_cli/stable_verification/test_score_submit.py
• README.md
• AGENTS.md

The Stable Compatibility Verifier is a specialized tooling suite within the Athena project designed to ensure that the server's behavior remains consistent with the official osu! stable client and legacy protocol expectations. It focuses on validating wire-format accuracy, request metadata safety, and response integrity across critical legacy endpoints such as score submission, replay downloading, and leaderboard retrieval.

This system acts as a quality gate, comparing internal implementation outputs against "golden" fixtures captured from official traffic or established reference implementations. It is partitioned into specific verifiers for different surfaces to maintain a modular monolith architecture while preserving externally observable stable-client behavior.

Sources: src/athena_cli/stable_verification/__init__.py, README.md:14-18, AGENTS.md:129-132

Architecture and Core Concepts

The verification system is organized around the concept of Surfaces, representing specific protocol areas (e.g., SCORE_SUBMIT, GETSCORES, REPLAY_DOWNLOAD). Each surface has associated VerificationStatus results and EvidenceTypes (e.g., GOLDEN_FIXTURE, AUTOMATED_TEST) to categorize the outcome of a compatibility check.

High-Level Component Interaction

The following diagram illustrates how the verification logic interacts with fixtures and internal mappers to produce results.

The diagram shows the flow from raw data and fixtures through specialized verification logic to final status results.
Sources: src/athena_cli/stable_verification/score_submit.py:44-60, tests/unit/athena_cli/stable_verification/test_replay_download.py:126-140

Score Submission Verification

The ScoreSubmitVerifier validates the integrity of the /web/osu-submit.php endpoint. It checks both the raw request metadata (to ensure no sensitive credentials like password hashes are leaked) and the structure of the chart response returned to the client.

Key Validation Rules

Verification involves checking for mandatory fields and ensuring safe data handling:

• Safety Check: The system explicitly forbids fragments like password_hash, session_token, and encrypted_payload in metadata reports to prevent security leaks.
• Response Integrity: Validates that the response includes required beatmap and overall chart fields such as rankBefore, rankAfter, accuracyAfter, and onlineScoreId.

Validation Category	Required Elements	Purpose
Request Metadata	content_type, field_order, score_field_count	Ensures request structure matches expectations.
Safety Policy	metadata-only	Prevents inclusion of raw secrets in verification artifacts.
Chart Fields	achieved, rankAfter, ppAfter, onlineScoreId	Ensures the client receives a valid result chart.

Sources: src/athena_cli/stable_verification/score_submit.py:34-42, src/athena_cli/stable_verification/score_submit.py:145-160, tests/unit/athena_cli/stable_verification/test_score_submit.py:42-55

Replay Download Diagnostics

Replay verification focuses on the integrity of .osr blobs and the /web/osu-getreplay.php route. It employs a diagnostic engine to determine why a replay might be failing or missing, providing detailed classifications like INTEGRITY_PASS, MISSING_SCORE, or STORAGE_INTEGRITY_FAILURE.

Diagnostic Sequence

The diagnostic process follows a strict hierarchy to isolate failures in the storage or database layer.

This sequence ensures that every layer of the replay retrieval process is validated for a specific score ID.
Sources: tests/unit/athena_cli/stable_verification/test_replay_download.py:431-455, tests/unit/athena_cli/stable_verification/test_replay_download.py:488-510

Leaderboard (Getscores) Verification

The GETSCORES surface is validated primarily through fixture-based testing of the /web/osu-osz2-getscores.php endpoint. The verifier ensures that the wire format conforms exactly to the stable client's expectations, including status codes and pipe-delimited formatting.

Wire Format Standards

• Status Mapping: Ranked maps must return status 2, while statuses like Loved (5) and Qualified (4) are also strictly enforced.
• Short Responses: Unsubmitted beatmaps or missing credentials must return the short body -1|false.
• Sanitization: Delimiters like pipes (|) and line breaks within artist or title names must be replaced to prevent breaking the delimited format.
• Framing: Responses must not contain HTTP chunk framing markers.

Sources: tests/unit/transports/web_legacy/test_getscores_fixtures.py:46-60, tests/unit/transports/web_legacy/test_getscores_fixtures.py:165-175, tests/unit/transports/web_legacy/test_getscores_fixtures.py:209-225

Data Models and Statuses

The verification system uses structured data models to report results.

Verification Result Model

Field	Type	Description
surface	StableSurface	The protocol area being verified (e.g., SCORE_SUBMIT).
status	VerificationStatus	Outcome: PASS, FAIL, UNAVAILABLE, or KNOWN_GAP.
scope	EvidenceScope	Whether the check is MANDATORY for compatibility.
diagnostic_summary	DiagnosticSummary	Detailed message and response byte size.
reference	str	Path to the fixture or source file used for comparison.

Sources: src/athena_cli/stable_verification/score_submit.py:236-249, tests/unit/athena_cli/stable_verification/test_score_submit.py:14-22

The Stable Compatibility Verifier ensures that Athena remains a drop-in replacement for official Bancho services by programmatically enforcing the "modular monolith" architecture's commitment to preserving externally observable stable-client behavior.