Background Jobs & QStash Queues
Relevant source files
The following files were used as context for generating this wiki page:
Background jobs in the HEX-YT-INTEL project are governed by the Hybrid Edge Symphony (ADR 005) architecture. This system is designed to bypass standard serverless execution limits by offloading high-latency tasks—specifically deep LLM synthesis—to specialized environments while maintaining strict security and data integrity.
The architecture utilizes a multi-cloud flow where Vercel acts as a secure "Bouncer" for authentication and quota enforcement, while background processing is handled via direct streaming or asynchronous persistence. This ensures that long-form content analysis (often exceeding 90 minutes) can be processed without encountering the standard 10-second serverless timeout limits.
Sources: GEMINI.md:126-136, CLAUDE.md:18-24, README.md:20-27
Hybrid Edge Architecture
The background processing lifecycle is split into three distinct operational phases to ensure reliability and security. This "Symphony" coordinates between Vercel, Cloudflare, and the database layer.
Core Components
| Component | Role | Responsibility |
|---|---|---|
| Vercel Bouncer | Gatekeeper | Authentication, HMAC signature verification, and atomic quota checks (~8s execution). |
| Cloudflare Worker | Processing Engine | High-performance LLM synthesis and Server-Sent Events (SSE) streaming (~58s execution). |
| S2S /persist | Data Integrity | Tamper-proof server-to-server endpoints for persisting analysis results using HMAC signatures. |
Sources: CLAUDE.md:18-24, README.md:20-27, AGENTS.md:16-24
Data Flow and Lifecycle
The following diagram illustrates the lifecycle of a background analysis job, from the initial user request to final data persistence.
The diagram shows the distributed flow between Vercel, Cloudflare, and Supabase for background tasks.
Sources: CLAUDE.md:8-24, GEMINI.md:126-136, AGENTS.md:26-38
Performance and Reliability Controls
To manage background jobs effectively, the system implements stratified timeouts and caching mechanisms.
Stratified Dual-Timeouts
The system utilizes specific timeout windows depending on the environment and task type:
- Connection Handshake: 3-second hard timeout to detect network faults early.
- Vercel Execution: 25-second maximum read for token streaming.
- Cloudflare Worker: Extended 90-second maximum read for intensive synthesis.
Sources: GEMINI.md:85-91, CLAUDE.md:11-14
Pre-Query Cache Hit Circuit
To minimize redundant background processing and reduce costs, the system implements a mandatory cache check. Before any analysis job is triggered, the analyses table is queried for matching video_id and user_id. If a result exists, the cached markdown is returned immediately, skipping the LLM generation queue.
Sources: GEMINI.md:80-83, CLAUDE.md:8-10
Monitoring and Observability
Background job health and system performance are monitored via an admin dashboard that tracks component status and latency.
Health Metrics
The system tracks the following components for operational readiness:
- Database: Status and latency check.
- Cloudflare Worker: Latency and error reporting for the streaming engine.
- Sentry: Configuration status for error tracking.
Sources: web/app/admin/dashboards/page.tsx:11-20, web/app/admin/dashboards/page.tsx:135-155
Monitoring Integration
| Service | Purpose | Link/Reference |
|---|---|---|
| Sentry | Error Tracking | Organizations/hex-tech-lab/projects |
| Vercel Monitoring | Edge Logs | hex-tech-lab/hex-yt-intel/monitoring |
| Supabase | Database Queries | project/_/editor |
Sources: web/app/admin/dashboards/page.tsx:210-235
Security and Verification
All background job results must be persisted via a secure, tamper-proof mechanism.
HMAC Signature Verification
The system uses verifyContentSig with a STREAM_HMAC_SECRET to ensure that data being saved to the database via the /persist endpoint actually originated from the trusted Cloudflare Worker engine and was not tampered with.
Sources: CLAUDE.md:23-24, AGENTS.md:30-34
Implementation Workflow
- Preparation: Validate
STREAM_HMAC_SECRETin the environment. - Execution: The Cloudflare Worker processes content and generates a signature.
- Persistence: The
/persistendpoint verifies the HMAC before writing to Supabase. - Synchronization: Frontend hooks (
useSSEStream.ts) manage the direct connection to the worker.
Sources: AGENTS.md:27-38
Summary
Background jobs in HEX-YT-INTEL are architected around the Hybrid Edge Symphony, ensuring that high-density content synthesis (UCIS v5.1) is reliable and secure. By combining Vercel's edge bouncer with Cloudflare's streaming capabilities and HMAC-signed persistence, the system handles complex, long-form background tasks without serverless timeout constraints.