Webhooks & Queue Consumers

Relevant source files

The following files were used as context for generating this wiki page:

• BY-STACK/webhooks.md
• claude-md-snippets/webhooks.md
• INTEGRATION.md
• MIGRATION-NOTES.md
• claude-md-snippets/universal.md
• BY-STACK/cron-jobs.md

Webhooks and queue consumers represent asynchronous entry points into the system where handlers execute concurrently with arbitrary invocations of themselves. In this environment, typical assumptions of sequential process execution are invalid; systems must account for "at-least-once" delivery, out-of-order messages, and duplicate requests.

The core architecture for these components focuses on identity verification, request deduplication, and ensuring atomicity between local state changes and external side effects. Key patterns like "Reserve-then-fill" and Check-and-Set (CAS) cursors are employed to maintain data integrity across distributed services such as Gmail, Stripe, Meta, and Google Calendar.

Sources: BY-STACK/webhooks.md:14-20, MIGRATION-NOTES.md:120-128

Reliability Patterns & Data Integrity

Reserve-then-Fill (Idempotency)

To prevent duplicate side effects (e.g., sending multiple SMS or creating multiple payment drafts), the system must perform a local database INSERT with a UNIQUE constraint or an idempotency key before calling any irreversible external API. If the insert fails due to an IntegrityError, the system should treat the request as a duplicate and return a success status (200 OK) to the caller.

Sources: claude-md-snippets/webhooks.md:3-5, BY-STACK/webhooks.md:33-66

The flow above ensures that external calls are only triggered if the local system has successfully claimed the unique message ID.
Sources: BY-STACK/webhooks.md:46-66

Atomic Cursor Updates (CAS)

When multiple webhooks compete for a shared cursor (e.g., Google Calendar sync_token), a Check-and-Set (CAS) pattern must be used. This involves updating the token only if the current value matches the expected old value.

Sources: claude-md-snippets/webhooks.md:7-9, BY-STACK/webhooks.md:76-103

Security & Verification

Verification of the caller is mandatory before any business processing begins. This includes signature validation and proper handling of client IP addresses.

Signature Validation

Handlers must use provider-specific verifiers (e.g., stripe.Webhook.construct_event or HMAC-SHA256) to ensure the payload hasn't been tampered with. It is critical to use raw bytes for validation rather than re-encoded text and to perform constant-time comparisons using hmac.compare_digest to prevent timing attacks.

Sources: claude-md-snippets/webhooks.md:11-13, BY-STACK/webhooks.md:139-148

Trusted Proxy Configuration

For rate-limiting decisions based on IP, the system must use a trusted proxy middleware (e.g., ProxyHeadersMiddleware). Direct access to headers like X-Forwarded-For is prohibited as they are easily spoofed by attackers to bypass rate limits.

Sources: claude-md-snippets/webhooks.md:15-17, BY-STACK/webhooks.md:120-136

Component	Protection Mechanism	Description
Rate Limiter	Trusted Proxy	Prevents X-Forwarded-For spoofing by validating proxy hop counts.
Signature Verifier	hmac.compare_digest	Prevents timing attacks during HMAC signature verification.
DB Layer	UNIQUE Constraint	Fundamental mechanism for idempotency in "Reserve-then-fill" patterns.
CAS Logic	rowcount check	Ensures cursor progression only happens once for a set of changes.

Sources: claude-md-snippets/webhooks.md:11-17, BY-STACK/webhooks.md:120-136

Integration with Queue Consumers

Queue consumers and scheduled tasks (Cron) often intersect with webhooks when they target the same database records. This creates potential race conditions between the scheduler and ad-hoc triggers.

Race Condition Mitigation

When both a beat scheduler and a webhook might trigger a task, use an atomic UPDATE to "claim" the record before processing.

# Atomic claim of a pending record
result = await session.execute(
    update(Messages)
    .where(Messages.id == msg_id, Messages.status == 'pending')
    .values(status='processing')
    .returning(Messages.id)
)
if result.rowcount == 0:
    return # Already claimed by another worker

Sources: BY-STACK/webhooks.md:162-171, BY-STACK/cron-jobs.md:104-114

Consumer Lifecycle Rules

• Terminal States: Every query must filter for rows that have not reached a terminal state to avoid infinite loops. Heuristics like attempts < MAX should be avoided in favor of explicit status transitions.
• Fail Closed: If a storage operation (like saving an OTP or session) fails, the side effect (like sending an SMS) must not proceed.
• Async/Sync Liveness: In async environments (FastAPI, Celery), blocking sync calls must be wrapped in asyncio.to_thread to prevent stalling the event loop.

Sources: claude-md-snippets/webhooks.md:21-23, BY-STACK/cron-jobs.md:27-56, BY-STACK/cron-jobs.md:124-132

Conclusion

The robustness of Webhooks & Queue Consumers relies on defensive programming against concurrent execution. By prioritizing local state persistence before external action and enforcing strict cryptographic and network-level verification, the system ensures at-least-once delivery does not degrade into uncontrolled side effects or security vulnerabilities.

Sources: BY-STACK/webhooks.md:14-21, MIGRATION-NOTES.md:120-125