Convex HTTP API & Mutations
Relevant source files
The following files were used as context for generating this wiki page:
The Convex HTTP API and Mutations system serves as the backbone for data persistence and external service integration within the bot-auto-ig project. This system provides a dual interface: a standard Mutation/Query interface for the React-based frontend and a specialized HTTP API (routed through .convex.site) used primarily by the Python automation runners and the Express server.
This architecture ensures that while the frontend interacts with Convex using its native reactive client, external services can perform secure, authenticated operations using standard HTTP protocols. The mutations ensure atomic updates to the underlying schema, maintaining consistency across workflow states, profile configurations, and scraping results.
System Architecture & Data Flow
The project utilizes Convex as a serverless backend. The interaction model is divided into public-facing mutations (typically used by the frontend with Clerk identity) and internal HTTP actions secured by an INTERNAL_API_KEY.
The diagram shows how different clients interact with the Convex backend via either standard mutations or HTTP actions.
Sources: convex/_generated/server.d.ts:25-80, server/shared/convexClient.ts:16-35, AGENTS.md:28-35
HTTP API Implementation
The HTTP API is implemented using Convex httpAction handlers. These are exposed via the .convex.site endpoint and allow external services to interact with the database using standard RESTful patterns.
Authentication and Security
External requests must include an Authorization header containing a Bearer token. This token is validated against the INTERNAL_API_KEY stored in the environment variables. If the key is missing or the token is incorrect, the system returns a 401 Unauthorized or 500 Internal Server Error.
Sources: server/shared/convexClient.ts:19-25, tests/convex/http.test.ts:18-40
Client-Side Integration (convexFetch)
The project provides a specialized TypeScript client for making these HTTP calls, featuring automated retries with exponential backoff for transient network issues.
| Feature | Description |
|---|---|
| Endpoint Transformation | Automatically converts .convex.cloud URLs to .convex.site for HTTP actions. |
| Retry Logic | Retries on HTTP 429, 5xx, and specific network errors (e.g., ECONNRESET, ETIMEDOUT). |
| Backoff Formula | baseDelay * 2^attempt + random jitter. |
Sources: server/shared/convexClient.ts:31-33, server/shared/convexClient.ts:89-130, server/shared/convexClient.ts:153-170
Workflow Mutations and State Management
Workflows are managed through a series of mutations that handle lifecycle transitions from idle to completed or failed.
Workflow Lifecycle Flow
The state machine for workflows is strictly enforced via transition assertions within the mutations.
Workflow state transitions managed by mutations in convex/workflows/mutations.ts.
Sources: convex/workflows/mutations.ts:133-245, convex/schema.ts:98-106
Key Mutation Functions
| Function | Purpose | Data Constraints |
|---|---|---|
create | Initializes a new workflow. | Requires name; sets status to idle. |
update | Modifies nodes/edges/config. | Forbidden if workflow status is running. |
start | Transitions workflow to pending. | Increments runsToday; checks maxRunsPerDay. |
updateStatus | Updates current node and state. | Updates startedAt or completedAt timestamps. |
remove | Deletes workflow and artifacts. | Deletes associated files from Convex storage. |
Sources: convex/workflows/mutations.ts:17-100, convex/workflows/mutations.ts:145-160
Data Schema & Persistence
The mutations interact with a structured schema defined in convex/schema.ts. This schema supports profiles, workflows, and scraping artifacts.
Core Tables
Entity relationship diagram illustrating the core Convex data model.
Sources: convex/schema.ts:5-150
Profile Management
Profiles are tracked with detailed metadata to manage scraping leases and health. HTTP routes like /api/profiles/sync-status allow the automation runtime to update these fields in real-time.
- Scrape Leases: Managed via
scrapeLeaseOwnerandscrapeLeaseExpiresAtto prevent concurrent access by multiple workers. - Limits:
dailyScrapingLimitandassignedAccountsLimitare enforced during mutations.
Sources: convex/schema.ts:12-34, server/shared/convexClient.ts:251-285
Conclusion
The Convex HTTP API and Mutations system provides a robust, type-safe, and authenticated bridge between the project's frontend, backend services, and Python automation runners. By centralizing state transitions in Convex mutations, the system ensures data integrity and atomic operations across the entire distributed architecture.