Anime Data Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Anime Data Pipeline automates the collection, processing, and visualization of personal anime viewing records. It integrates data from the Bangumi API, processes it through Python backend scripts, and delivers content to a React-based frontend using CDN-hosted static assets.
Sources: README.md:19-27, blog/2025/12/14/01_博客更新日志和计划.md:23-53
System Architecture
The pipeline consists of three primary layers: the Backend Data Acquisition layer, the Static Storage and CDN layer, and the Frontend Visualization layer.
The architecture separates data acquisition from presentation to ensure high performance and reduce direct API calls to Bangumi during user browsing.
Sources: README.md:27, py/anime/Api.py:46-51, src/utils/cdn/linkJsDelivr.ts:25-30
Backend Data Acquisition
The backend utilizes the Api class to synchronize viewing status and metadata from Bangumi.
Data Fetching and Rate Limiting
Python scripts request data from https://api.bgm.tv/v0. An ApiReqRateLimiter ensures requests stay within acceptable limits (set to 119 in the implementation). The system fetches user collections, specific subject details, characters, voice actors (CVs), and related entries.
Sources: py/anime/Api.py:28-40, py/anime/Api.py:160-165
Image Processing
The system employs a multi-threaded downloading system to acquire anime posters, character images, and actor portraits.
- Queue Management: A
queue.Queuestores tuples of(type, id, url). - Concurrency: Eight worker threads process the queue simultaneously.
- Storage: Images are saved in the
./data/{type}/{id}.jpgdirectory structure.
Sources: py/anime/Api.py:86-114, py/anime/Api.py:53-73
Backend Components Summary
| Function | Description | Source |
|---|---|---|
requires | Refreshes the full dataset for a specific user. | py/anime/Api.py:75-81 |
_get_user_watching_anime | Syncs viewing status (Watched, Watching, etc.) and comments. | py/anime/Api.py:133-205 |
_get_anime_characters | Retrieves character lists and links them to Voice Actors (Actors). | py/anime/Api.py:228-259 |
_get_relations | Fetches related anime (e.g., Prequels, Sequels). | py/anime/Api.py:261-282 |
download_all_img | Downloads all assets via a multi-threaded queue. | py/anime/Api.py:83-114 |
Data Model & Storage
Data is serialized into JSON format for static hosting. The project primarily uses two files: ANiMeRecord.json and Actor.json.
Relationship Schema
Storage Locations
- JSON Path:
./data/ANiMeRecord.jsonand./data/Actor.json. - Image Path:
./data/anime/,./data/kyara/,./data/cv/, and./data/relation/.
Sources: py/anime/Api.py:32-34, py/anime/Api.py:339-366
Frontend Visualization
The frontend is built using React and Docusaurus, loading data through CDN utility functions.
CDN Integration
The linkJsDelivr.ts utility resolves asset paths to CDN mirrors (jsDelivr).
- Anime Engine: Dedicated engine for
HXLoLi-ANiMerepository. - URL Resolution:
toAnimeCdnUrlmaps local relative paths to CDN URLs using tags or commit IDs.
Sources: src/utils/cdn/linkJsDelivr.ts:98-100, src/utils/cdn/linkJsDelivr.ts:153-157
Visualization Modules
- List View: Groups anime by year and quarter (Spring, Summer, Autumn, Winter). It supports filtering by viewing status (e.g., "Watching") and sorting by last update or broadcast date.
- Detail View: Displays comprehensive metadata, episode lists (formatted by type such as SP or OP), and character/actor cards.
- Graph View: Uses a force-directed graph to visualize the connections between Anime, Characters, and Actors.
Sources: src/pages/anime/index.tsx:50-91, src/pages/anime/details.tsx:162-250, src/pages/anime/graph.tsx:75-90
Viewing Status Types
The frontend maps WatchStatus enums to readable Chinese labels:
WANT_TO_WATCH: 想看WATCHED: 看过WATCHING: 在看ON_HOLD: 搁置DROPPED: 抛弃
Sources: src/pages/anime/details.tsx:21-34
Data Synchronization Sequence
The following diagram illustrates the synchronization process triggered by the backend script:
Sources: py/anime/Api.py:75-81, py/anime/Api.py:339-383
Conclusion
The Anime Data Pipeline provides a robust framework for personal anime tracking by decoupling the Bangumi data source from the user interface. By utilizing multi-threaded backend scrapers and a static CDN-based frontend delivery system, the project achieves an efficient, visually rich, and low-latency experience for displaying anime viewing records and relationship图谱.
Sources: README.md:27, blog/2025/12/14/01_博客更新日志和计划.md:73-77