Relevant source files

The following files were used as context for generating this wiki page: - [crates/metadata/src/repos/gc.rs](https://github.com/JinxCappa/cellar/blob/main/crates/metadata/src/repos/gc.rs) - [crates/server/src/handlers/admin.rs](https://github.com/JinxCappa/cellar/blob/main/crates/server/src/handlers/admin.rs) - [crates/server/src/main.rs](https://github.com/JinxCappa/cellar/blob/main/crates/server/src/main.rs) - [crates/cli/src/main.rs](https://github.com/JinxCappa/cellar/blob/main/crates/cli/src/main.rs) - [crates/metadata/src/postgres.rs](https://github.com/JinxCappa/cellar/blob/main/crates/metadata/src/postgres.rs) - [crates/metadata/src/store.rs](https://github.com/JinxCappa/cellar/blob/main/crates/metadata/src/store.rs) - [README.md](https://github.com/JinxCappa/cellar/blob/main/README.md)

# Garbage Collection System The Garbage Collection (GC) system in Cellar is a multi-layered background service responsible for maintaining storage efficiency and data integrity. Its primary purpose is to reclaim disk space by removing unreferenced or orphaned data artifacts—such as chunks, manifests, and expired upload sessions—while ensuring that live data remains protected during active operations. The system is designed for multi-tenant isolation, allowing GC jobs to be scoped to specific caches or run globally across the entire server. It features a robust architecture including an automatic scheduler, manual triggers via administrative APIs, and a watchdog mechanism to recover from crashes or hangs. Sources: [README.md:215-225](), [crates/server/src/main.rs:215-230](), [crates/server/src/handlers/admin.rs:585-600]() ## GC Job Architecture The GC system operates using a job-based model where each task is tracked in the metadata database. This allow for monitoring, resumption, and prevention of concurrent conflicting operations. ### Job Lifecycle and States A GC job progresses through a defined set of states. Conflicting jobs of the same type for the same cache are prevented by database-level unique constraints on the `gc_jobs` table. ```mermaid stateDiagram-v2 [*] --> Queued: Triggered (API/Auto) Queued --> Running: Process Spawned Running --> Finished: Success (Errors = 0) Running --> Failed: Error Occurred Running --> Failed: Watchdog (Panic/Crash) Queued --> Failed: Watchdog (Server Restart) Finished --> [*] Failed --> [*] ``` The diagram shows the transition states of a GC job from creation to completion or failure. Sources: [crates/metadata/src/repos/gc.rs:65-75](), [crates/server/src/handlers/admin.rs:630-650](), [crates/server/src/main.rs:340-360]() ### Job Types The system supports several specialized job types, each targeting specific categories of data: | Job Type | Description | Targeted Artifacts | | :--- | :--- | :--- | | `upload_gc` | Cleans up abandoned or expired upload sessions. | `upload_sessions`, `upload_expected_chunks` | | `chunk_gc` | Removes unreferenced chunks (refcount = 0) older than the grace period. | `chunks`, storage objects | | `manifest_gc` | Cleans up orphaned manifests and their associated chunk mappings. | `manifests`, `manifest_chunks` | | `committing_gc` | Recovers sessions stuck in 'committing' state by rolling back refcounts. | Stuck `upload_sessions` | | `storage_sweep` | Scans physical storage for orphaned files with no corresponding metadata. | Orphaned storage files | Sources: [crates/metadata/src/repos/gc.rs:45-60](), [crates/server/src/handlers/admin.rs:850-1000]() ## Operational Logic ### Automated Scheduling Cellar includes an automatic GC scheduler that can be enabled via configuration. When active, it iterates through all configured caches (including the public cache) at a specified interval to trigger the requested GC jobs. ```mermaid flowchart TD Start([Interval Timer]) --> ListCaches[List All Caches] ListCaches --> ForEachCache{For Each Cache} ForEachCache --> ForEachType{For Each Job Type} ForEachType --> CheckRunning{Job Running?} CheckRunning -- No --> Spawn[Spawn Background Task] CheckRunning -- Yes --> Skip[Skip Type] Spawn --> ForEachType Skip --> ForEachType ForEachType -- Done --> ForEachCache ForEachCache -- Done --> End([Wait for Next Interval]) ``` The flow demonstrates how the automatic scheduler triggers jobs across multiple tenants and types. Sources: [crates/server/src/main.rs:215-275]() ### Data Protection and Safety The GC system implements several safety mechanisms to prevent data loss: 1. **Grace Period**: Chunks and manifests are only eligible for deletion if they have zero references and are older than a configured `grace_period_secs` (e.g., 3600s). 2. **Active Session Awareness**: `chunk_gc` and `storage_sweep` check `upload_sessions` to ensure chunks currently being uploaded (which have a refcount of 0 until commit) are not deleted. 3. **Atomic Deletion**: In PostgreSQL, the system uses Common Table Expressions (CTEs) with `FOR UPDATE` to lock and delete metadata rows atomically, preventing TOCTOU (Time-of-Check to Time-of-Use) races where a refcount might be incremented just as the GC attempts a delete. Sources: [crates/server/src/handlers/admin.rs:1050-1100](), [crates/metadata/src/postgres.rs:720-750](), [README.md:285-295]() ## Resumable Storage Sweep The `storage_sweep` job is unique because it must iterate over potentially millions of objects in a storage backend (like S3 or local disk). To handle this, it implements a checkpointing system. ### Checkpoint Mechanism During a sweep, the system stores a `SweepCheckpoint` in the database. This checkpoint includes: - **Continuation Tokens**: Page-based tokens for resumable listings (specific to the storage backend). - **Processed Counters**: Number of chunks and manifests processed. - **Timestamp**: Last time the checkpoint was updated. If the server restarts or the job fails, the next `storage_sweep` can resume from the last saved page rather than restarting from the beginning. Sources: [crates/server/src/handlers/admin.rs:25-60](), [crates/server/src/handlers/admin.rs:1350-1450]() ## Monitoring and Administration ### Management via CLI The `cellarctl` tool provides commands to trigger and monitor GC jobs. ```bash # Run a specific job type for a specific cache cellarctl gc run --job-type chunk_gc --cache-id --wait # View recent GC job history cellarctl gc status --limit 10 ``` Sources: [crates/cli/src/main.rs:225-245](), [README.md:195-210]() ### Metrics and Statistics GC jobs report their progress via the `GcStats` structure, which includes: - `items_processed`: Total count of items checked. - `items_deleted`: Number of items actually removed. - `bytes_reclaimed`: Amount of storage space recovered. - `errors`: Count of non-fatal errors encountered during the job. These statistics are persisted in the database and visible via the API and CLI. Sources: [crates/metadata/src/repos/gc.rs:80-95](), [crates/server/src/handlers/admin.rs:720-740]() ## Crash Recovery and the Watchdog The system handles server failures through two distinct mechanisms: 1. **Startup Recovery**: On server boot, the function `recover_orphaned_gc_jobs` queries for any jobs stuck in `queued` or `running` states and marks them as `failed` with a recovery error. 2. **Runtime Watchdog**: A dedicated `GcTaskRegistry` monitors active Tokio handles. If a task panics or ends unexpectedly without updating its database state, the watchdog detects the missing handle and marks the job as `failed`. Sources: [crates/server/src/main.rs:340-380](), [crates/server/src/main.rs:200-210]() ## Conclusion The Garbage Collection System in Cellar is a critical component for maintaining a healthy binary cache. By combining multi-tenant job isolation, atomic database operations, and robust crash recovery mechanisms, it provides safe and efficient storage management suitable for both local development and high-volume production environments. assistant

Relevant source files