Message Ingestion & Processing

Relevant source files

The following files were used as context for generating this wiki page:

• src/index.ts
• src/api/update.ts
• AGENTS.md
• README.md
• tests/integration/comprehensive-e2e.test.ts
• tests/telegram-integration.test.ts

Message Ingestion & Processing refers to the end-to-end lifecycle of receiving Telegram updates, validating their source, extracting text data, and persisting messages for historical analysis. This system functions as a Cloudflare Worker that handles incoming webhooks and routes data to specialized storage and analysis modules.

The system ensures that only valid, non-bot messages are recorded to Cloudflare KV storage with a specific Time-To-Live (TTL). It serves as the data foundation for downstream features such as daily summaries, active user tracking, and content analysis. Sources: README.md:1-15, AGENTS.md:1-5

Webhook Entry Point

The fetch handler in the main entry point receives HTTPS POST requests from Telegram. It validates the request using a bot token in the URL path and a secret token in the X-Telegram-Bot-Api-Secret-Token header. Sources: src/index.ts:40-192

Authentication Flow

1. The Worker receives a POST request at /tg/{TOKEN}/webhook.
2. It compares the {TOKEN} in the path against the env.TOKEN secret.
3. It validates the X-Telegram-Bot-Api-Secret-Token header against env.SECRET.
4. If either check fails, the Worker returns a 403 Forbidden response, unless the environment is set to development.

The following diagram illustrates the entry point logic:

Message Extraction and Filtering

The system employs strict filtering to prevent irrelevant data from entering the storage layer. The getTextMessage function extracts message objects while applying several exclusion rules. Sources: src/api/update.ts, tests/telegram-integration.test.ts:241-285

Rule	Description
Bot Exclusion	Messages where from.is_bot is true are discarded.
Content Check	Messages without a text field (e.g., pure photos or stickers) are discarded.
Forward Filtering	Messages containing forward_origin or forward_date are not recorded to history.
Command Identification	Messages starting with / are identified as commands and processed but not analyzed for violations.

Sources: src/index.ts:187-195, tests/telegram-integration.test.ts:241-285, tests/integration/comprehensive-e2e.test.ts:211-220

Persistence Layer

Validated messages are persisted using a dual-storage strategy involving Cloudflare KV for raw history and Durable Objects for real-time counters. Sources: README.md:10-18, AGENTS.md:143-146

KV Storage Schema

Messages are stored in the HISTORY KV namespace. Keys follow a specific pattern to enable time-based retrieval:
msg:{chat}:{ts}:{id}

• chat: The Telegram Chat ID.
• ts: The message timestamp.
• id: The unique Message ID.
• TTL: All keys are set with a TTL of 604,800 seconds (7 days).

Sequence of Persistence

The following sequence diagram shows how a message is recorded after validation:

Message Handling Logic

Once recorded, the message is passed to handleUpdate for asynchronous processing. This function determines if the message is a user command or a regular chat message requiring analysis. Sources: src/index.ts:192-194, src/api/update.ts

1. Command Processing: If the message text starts with /, the handleUpdate function routes it to specific command handlers (e.g., /summary, /top, /help).
2. Content Analysis: If it is a regular message, the system performs background analysis for profanity or criminal code violations using the ViolationHandler.
3. Admin Routing: Requests to the /admin path bypass standard message ingestion and are handled by handleAdminRequest.

Sources: src/index.ts:46-60, tests/integration/comprehensive-e2e.test.ts:200-210, tests/telegram-integration.test.ts:125-145

Technical Specifications

Component	Technology	Role
Runtime	Cloudflare Workers	Serverless execution environment.
History Store	Cloudflare KV	Ephemeral storage for chat history (7 days).
State Store	Durable Objects	Atomic counters for users and message blocks.
Database	Cloudflare D1	Permanent storage for violation history and processed summaries.
API	Telegram Webhook	Inbound message delivery via HTTPS POST.

Sources: AGENTS.md:1-25, README.md:5-25, src/index.ts:1-25

Conclusion

The Message Ingestion & Processing system provides a robust gateway for Telegram data. By combining Cloudflare Workers' edge execution with KV and Durable Objects, the bot achieves low-latency message recording and reliable filtering. This architecture ensures that historical data is sanitized, authenticated, and available for the bot's advanced summarization and moderation features. Sources: README.md:1-50, AGENTS.md:140-150