Relevant source files

The following files were used as context for generating this wiki page: - `src/content/config.ts` - `public/admin/config.yml` - `migration/process_content.py` - `supabase/functions/search-content/index.ts` - `lou_gehrig_fan_club_master_build_v5.md` - `CMS_README.md` - `PROJECT_STATUS_ASSESSMENT.md` - `SEARCH_CRON_FINAL_STATUS.md`

# Data Flow The data flow within the LGFC-WEBAPP is characterized by two primary streams: automated content ingestion from external sources and manual content management via a headless CMS. Both streams converge into a central Supabase database, which serves as the single source of truth for the Astro-based frontend. This architecture ensures a constant influx of relevant, fresh content while providing administrators with robust tools to curate and manage the community's data. The automated flow is handled by a `search-cron` job that runs hourly, fetching data from RSS feeds and APIs, processing it, and storing it in the database. The manual flow is managed through the Decap CMS, allowing administrators to create and edit structured content like news, events, and milestones, which are stored as Markdown files in the Git repository and then rendered by the frontend. User interaction data, such as votes, is captured directly by the frontend and sent to the Supabase backend. Sources: [`lou_gehrig_fan_club_master_build_v5.md`](), [`SPACES_PROJECT_SUMMARY.md`]() ## High-Level Data Flow This diagram illustrates the two main pathways for data entering the application ecosystem: automated ingestion and manual content management. ```mermaid graph TD subgraph Automated Ingestion A[External Sources
RSS Feeds, NYT API] --> B{GitHub Actions
search-cron}; end subgraph Manual Management C[Administrator] --> D{Decap CMS}; D --> E[Git Repository
Markdown Files]; end subgraph Application Backend F[Supabase Database]; end subgraph Frontend G[Astro Web Application]; end B --> F; E --> G; C -.-> F; G <--> F; style A fill:#f9f,stroke:#333,stroke-width:2px style D fill:#ccf,stroke:#333,stroke-width:2px ``` Sources: [`lou_gehrig_fan_club_master_build_v5.md`](), [`CMS_README.md`]() ## Automated Content Ingestion (`search-cron`) The `search-cron` system is a GitHub Actions workflow responsible for automatically discovering and ingesting new content related to Lou Gehrig from various online sources. It runs hourly, ensuring the website's content remains current. Sources: [`lou_gehrig_fan_club_master_build_v5.md`]() ### Ingestion Workflow The workflow is defined in `.github/workflows/search-cron.yml` and consists of several steps that fetch, process, and store data. ```mermaid flowchart TD A[Schedule: Hourly Cron] --> B{Start GitHub Action}; B --> C[Checkout Code]; C --> D[Install Dependencies]; D --> E[Check DB Migrations
scripts/check_migrations.mjs]; E --> F[Run Diagnostic Test
scripts/test_ingest.mjs]; F --> G{Run Ingestion Script
scripts/ingest.mjs}; G -- Fetches from --> H[RSS Feeds & NYT API]; G -- Processes & Deduplicates --> I[Content Items]; I -- Inserts into --> J[Supabase Table
content_items]; ``` This diagram shows the sequence of the `search-cron` GitHub Action, from its scheduled trigger to the final insertion of data into the Supabase database. Sources: [`lou_gehrig_fan_club_master_build_v5.md`]() ### Key Components | Component | Description | Source File | | :--- | :--- | :--- | | `search-cron.yml` | The GitHub Actions workflow file that defines the cron schedule and job steps for content ingestion. | `lou_gehrig_fan_club_master_build_v5.md` | | `ingest.mjs` | The core Node.js script that connects to external sources, fetches content, and performs the insertion into Supabase. | `lou_gehrig_fan_club_master_build_v5.md` | | `content_items` | The Supabase table where ingested articles and posts are stored. It includes fields for title, content, source URL, and a content hash for deduplication. | `lou_gehrig_fan_club_master_build_v5.md` | | `search_state` | A Supabase table used to manage the state of the cron job between runs. | `PROJECT_STATUS_ASSESSMENT.md` | The connection to Supabase from the scripts is made using a public API key, with access control managed by Row Level Security (RLS) policies. Sources: [`SEARCH_CRON_FINAL_STATUS.md`]() ## Content Management System (CMS) Flow The Decap CMS provides a user-friendly interface for administrators to manage structured content. This content is stored as Markdown files within the Git repository, which are then used by Astro to build static pages. Sources: [`CMS_README.md`]() ### CMS Architecture The CMS data flow involves the administrator, the CMS UI, the Git repository, and the live website. ```mermaid sequenceDiagram participant Admin as "Administrator" participant CMS as "Decap CMS UI (/admin)" participant Repo as "GitHub Repository" participant Netlify as "Netlify Build" participant Site as "Live Website" Admin->>CMS: Logs in via Netlify Identity Admin->>CMS: Creates/Edits a 'News' article CMS->>Repo: Commits new/updated .md file to /src/content/news/ Repo->>Netlify: Triggers a new build via webhook Netlify->>Repo: Pulls latest changes Netlify->>Netlify: Runs 'astro build' Netlify->>Site: Deploys new static files Note over Admin, Site: New content is live ``` This sequence shows how an administrator's action in the CMS results in an updated live website. Sources: [`CMS_README.md`](), [`lou_gehrig_fan_club_master_build_v5.md`]() ### Content Collections The CMS is configured in `public/admin/config.yml` to manage several types of content, known as "collections." Each collection has a defined schema that dictates its fields. | Collection | Description | Storage Path | | :--- | :--- | :--- | | `Milestones` | Historical events and important dates. | `src/content/milestones/` | | `News Articles` | Club news and updates. | `src/content/news/` | | `Events` | Upcoming and past fan club or MLB events. | `src/content/events/` | | `Memorabilia` | Catalog of memorabilia items. | `src/content/memorabilia/` | | `Photos` | Photo gallery content. | `src/content/photos/` | | `Books` | Book reviews and recommendations. | `src/content/books/` | Sources: [`CMS_README.md`]() Below is an example snippet from a CMS configuration defining the fields for the `Events` collection. ```yaml # From public/admin/config.yml (conceptual) - name: "events" label: "Events" folder: "src/content/events" create: true fields: - { label: "Title", name: "title", widget: "string" } - { label: "Start Date", name: "startDate", widget: "datetime" } - { label: "End Date", name: "endDate", widget: "datetime", optional: true } - { label: "Location", name: "location", widget: "string", optional: true } - { label: "Body", name: "body", widget: "markdown" } - { label: "Is Club Event", name: "isClubEvent", widget: "boolean", default: true } ``` Sources: [`CMS_README.md`]() ## Database Schema The Supabase PostgreSQL database is the central repository for all dynamic application data, including user information, ingested content, and user interactions like votes. ### Entity Relationship Diagram This diagram shows the relationships between the core tables in the Supabase database. ```mermaid erDiagram members { UUID user_id PK string full_name string email string preferred_screen_name bool is_admin } content_items { INTEGER id PK string title string content_text string source_url string content_hash UUID created_by FK } media_files { INTEGER id PK string media_url string media_type string file_name } votes { INTEGER id PK UUID user_id FK INTEGER media_file_id FK datetime timestamp } voting_rounds { INTEGER id PK string status } round_winners { INTEGER id PK INTEGER round_id FK INTEGER media_file_id FK } members ||--o{ content_items : "creates" members ||--o{ votes : "casts" media_files ||--o{ votes : "receives" voting_rounds ||--o{ round_winners : "has" media_files ||--o{ round_winners : "is" ``` Sources: [`lou_gehrig_fan_club_master_build_v5.md`](), [`PROJECT_STATUS_ASSESSMENT.md`]() ### Key Tables - **`members`**: Stores user profile information. Integrated with Supabase Auth, so `user_id` is a foreign key to the `auth.users` table. - **`content_items`**: Holds content ingested by the `search-cron` job as well as potentially user-generated posts. - **`media_files`**: Contains metadata about media, such as images for the voting system. The actual files are intended to be stored in Backblaze B2. - **`votes`**: Records each individual vote cast by a member on a media file. - **`voting_rounds`** & **`round_winners`**: Manages the 24-hour voting cycle, including tracking winners and handling ties. Sources: [`lou_gehrig_fan_club_master_build_v5.md`](), [`UPDATED_PROJECT_PLAN_WITH_NOTES.md`]() ## Summary The data flow of the LGFC-WEBAPP is a robust, dual-channel system designed for both scalability and ease of management. The automated `search-cron` ensures a continuous stream of fresh, relevant content with minimal human intervention. Simultaneously, the Decap CMS empowers administrators to precisely control and curate the static and dynamic content presented to users. All data funnels into a well-structured Supabase database, which provides a reliable backend for the frontend application and user interactions, creating a rich and dynamic experience for the fan club members.