Blob Object Storage
Relevant source files
The following files were used as context for generating this wiki page:
The Blob Object Storage system in Athena provides a content-addressed storage layer designed for streaming writes, integrity verification via SHA-256 metadata, and automatic deduplication. It serves as the primary mechanism for persisting large binary objects, such as .osu beatmap files, while ensuring that identical content is only stored once across the system.
The system is partitioned into a high-level coordination service (BlobStorageService), infrastructure backends (e.g., LocalBlobStorageBackend), and domain-driven metadata models. This architecture allows the application to remain agnostic of the underlying storage medium, supporting local filesystem storage while providing the necessary hooks for horizontal scaling and different backend implementations.
Sources: src/osu_server/services/commands/storage/blob_storage.py:1-40, README.md
System Architecture
The storage architecture follows a hexagonal design where the core service coordinates between a storage backend and a metadata repository.
Component Overview
| Component | Responsibility |
|---|---|
| BlobStorageService | Coordinates stream writes, calculates SHA-256 hashes, handles deduplication logic, and manages metadata persistence. |
| BlobStorageBackend | Interface for raw byte persistence. Handles staging, atomicity, and chunked reading. |
| LocalBlobStorageBackend | A concrete implementation using the local filesystem with atomic renames and content-addressed paths. |
| BlobQueryRepository | Interface for looking up existing blob metadata to support deduplication. |
| UnitOfWork | Ensures atomic updates between the storage backend (finalized files) and the metadata database. |
Sources: src/osu_server/services/commands/storage/blob_storage.py:44-60, src/osu_server/infrastructure/storage/local.py:31-40
Logic Flow: Staged Write and Deduplication
The system uses a "stage-then-commit" pattern to ensure data integrity. Content is written to a temporary location while its SHA-256 hash is calculated in real-time. If the hash matches an existing record, the temporary file is discarded to save space.
This diagram illustrates the deduplication and staging logic implemented in the service.
Sources: src/osu_server/services/commands/storage/blob_storage.py:77-130
Implementation Details
Content-Addressed Storage Keys
Athena uses a specific directory structure for local storage to prevent directory performance degradation. Storage keys are derived from the SHA-256 digest in the format: sha256/{prefix1}/{prefix2}/{full_digest}.
- Example:
sha256/e3/b0/e3b0c442... - Validation: The backend validates that the storage key matches the pattern and that the path does not escape the storage root.
Sources: src/osu_server/infrastructure/storage/local.py:183-205, src/osu_server/services/commands/storage/blob_storage.py:176-177
Local Backend Operations
The LocalBlobStorageBackend implements atomic writes using a two-step process:
- Staging: Files are created in a
.tmpdirectory within the storage root usingtempfile.mkstemp. - Finalization: The
os.linkcommand is used to create a hard link from the temporary file to the final content-addressed path, ensuring atomic publication.
Sources: src/osu_server/infrastructure/storage/local.py:108-150
Data Models
Blob Entity
The Blob model represents a successfully persisted object. It is an immutable dataclass.
| Field | Type | Description |
|---|---|---|
id | int | Unique database identifier. |
sha256 | str | 64-character lowercase hex string. |
byte_size | int | Non-negative size of the content. |
content_type | str | MIME type (e.g., application/x-osu-beatmap). |
storage_backend | str | Identifier for the backend (e.g., "local"). |
storage_key | str | The backend-specific key. |
Sources: tests/unit/domain/test_blob.py:10-40, tests/unit/repositories/test_blob_repository_contract.py:53-65
Integration with Beatmap System
The FetchBeatmapFileUseCase utilizes the Blob Object Storage to store .osu files fetched from external providers.
This sequence shows how the beatmap fetch process interacts with storage to persist file bodies.
Sources: src/osu_server/services/commands/beatmaps/fetch.py:133-228, tests/unit/factories/test_beatmap_blob_contract.py:68-85
Source Verification
When a file is stored via the beatmap fetch use-case, it is wrapped in a BeatmapFileAttachment. This attachment links the Blob to a specific beatmap_id and records metadata such as the source (e.g., OSU_CURRENT, COMMUNITY_MIRROR) and original_filename.
Sources: src/osu_server/domain/beatmaps.py, src/osu_server/services/commands/beatmaps/fetch.py:228-243
Error Handling
The system defines specific exceptions for storage failures:
- BlobContentTypeError: Raised if a write is attempted without a MIME type.
- BlobContentUnavailableError: Raised when a read is requested for a blob that exists in metadata but is missing from the physical backend.
- BlobStorageWriteError: Raised for fatal persistence failures, such as metadata creation races.
- BackendWriteError: Infrastructure-level error for filesystem issues (e.g., disk full, permission denied).
Sources: src/osu_server/services/commands/storage/blob_storage.py:30-40, src/osu_server/infrastructure/storage/errors.py
Summary
Athena's Blob Object Storage provides a robust, content-addressed persistence layer. By separating metadata from physical storage and enforcing SHA-256 integrity, the system ensures that large assets like beatmap files are stored efficiently and reliably. The use of atomic filesystem operations and a "stage-then-commit" workflow minimizes the risk of partial writes or corrupted data.