Voice Interaction (Twilio & ElevenLabs)

Relevant source files

The following files were used as context for generating this wiki page:

• convex/twilio.ts
• convex/ai.ts
• README.md
• convex/http.ts
• agent_configs/TalkCRM.json
• PROJECT_DESCRIPTION.md

The Voice Interaction system is the core feature of TalkCRM, enabling sales representatives to interact with their Salesforce CRM hands-free using natural language over a standard phone call. This system integrates Twilio for telephony services and ElevenLabs for conversational AI, orchestrated by a Convex backend. The primary goal is to allow users to perform CRM operations like searching records, updating opportunities, and creating tasks by simply speaking.

The architecture is designed to be voice-first, where a user calls a dedicated Twilio number, their call is routed to an ElevenLabs AI agent, and that agent interacts with the Convex backend via HTTP webhooks to perform Salesforce actions. The entire experience is conversational, with the AI providing spoken feedback and confirmations.

Sources: README.md, PROJECT_DESCRIPTION.md

Architecture Overview

The system's architecture connects the user (sales rep on a phone) to Salesforce through a series of services. Twilio handles the phone call, which is connected to an ElevenLabs Conversational AI agent via a WebSocket. The ElevenLabs agent processes the user's speech and, when necessary, calls tools hosted as HTTP actions on the Convex backend. Convex then communicates with the Salesforce REST API to execute the requested operations.

Sources: README.md, PROJECT_DESCRIPTION.md, convex/http.ts

This diagram illustrates the high-level data flow of the voice interaction system.

Twilio Integration

Twilio provides the telephony infrastructure, including the phone number that users call and the connection to the voice AI agent.

Sources: convex/twilio.ts, convex/http.ts

Incoming Call Handling

When a user calls the dedicated TalkCRM number, Twilio sends an HTTP POST request to the /webhooks/twilio/incoming endpoint on the Convex backend. This endpoint is the primary entry point for all voice interactions.

The handler performs the following steps:

1. Caller ID Lookup: It extracts the caller's phone number (From field) and uses it to look up the user in the TalkCRM database via internal.users.getUserByPhoneInternal.
2. User Authentication:
   • If the caller's number is not registered, the call is rejected with a spoken message instructing them to sign up.
   • If the user's account is not active, the call is rejected with a message indicating their account status.
3. Conversation Logging: If the user is authenticated, a new conversation record is created in the database, linking the callSid to the userId.
4. Connect to ElevenLabs: The endpoint returns a TwiML (Twilio Markup Language) response that instructs Twilio to connect the call to the ElevenLabs Conversational AI agent via a WebSocket.

Sources: convex/http.ts:601-649

The following sequence diagram shows the incoming call and authentication flow.

TwiML for ElevenLabs Connection

The TwiML response passes user-specific context (user_id, caller_phone, user_name) to the ElevenLabs agent as dynamic variables. This context is crucial for authenticating subsequent API calls from the agent to the backend.

<!-- Source: convex/http.ts:636-642 -->
<Response>
  <Say voice="alice">Welcome back, {userName}. Connecting you to your assistant.</Say>
  <Connect>
    <ConversationalAi url="wss://api.elevenlabs.io/v1/convai/conversation?agent_id=${agentId}&amp;dynamic_variables=${dynamicVars}" />
  </Connect>
</Response>

Sources: convex/http.ts:636-642

Outbound Calls

The system also includes a makeOutboundCall action, which can initiate a call from Twilio to a specified number and connect it to an ElevenLabs agent. This is marked for future use, such as implementing a call-back feature.

Sources: convex/twilio.ts:114-169

ElevenLabs Integration

ElevenLabs provides the Conversational AI agent that interacts with the user. It handles Automatic Speech Recognition (ASR), Natural Language Understanding (NLU), and Text-to-Speech (TTS).

Sources: agent_configs/TalkCRM.json, convex/http.ts

Agent Configuration

The agent's personality, prompt, and technical settings are defined in agent_configs/TalkCRM.json. The agent is configured to be a "high-energy voice assistant" that acts as a "hype partner" for the sales rep.

Key configuration parameters include:

Parameter	Value	Description
tts.model_id	eleven_turbo_v2	The text-to-speech model used for the agent's voice.
tts.voice_id	mHV5m7DLaQM0bIAP6BTK	The specific voice used for the agent.
turn.turn_timeout	10	Seconds of silence before the agent considers the user's turn to be over.
agent.first_message	"Hey! I'm TalkCRM..."	The initial greeting spoken to the user.
agent.prompt	(See file)	A detailed system prompt defining the agent's personality, tools, and behavior.
dynamic_variables	caller_phone, user_id, user_name	Placeholders for user context passed from Twilio.

Sources: agent_configs/TalkCRM.json

Tool Calling

The agent is empowered with a set of "tools" it can use to interact with Salesforce. These tools are implemented as HTTP endpoints in the Convex backend. The primary tool is the intelligent assistant tool.

Tool Name	Endpoint	Description
assistant	/tools/assistant	The primary intelligent endpoint that interprets natural language to perform any Salesforce action.
search_salesforce	/tools/search	(Legacy) Finds accounts, contacts, opportunities, etc.
create_record	/tools/create-record	(Legacy) Creates new Salesforce records.
update_record	/tools/update-record	(Legacy) Updates existing Salesforce records.
log_call	/tools/log-call	(Legacy) Logs a call activity.
get_my_tasks	/tools/my-tasks	(Legacy) Retrieves the user's open tasks.
get_my_pipeline	/tools/my-pipeline	(Legacy) Retrieves the user's open opportunities.

Sources: README.md, convex/http.ts:114-411, agent_configs/TalkCRM.json

When the agent determines it needs to perform a Salesforce action, it makes a POST request to the corresponding endpoint, such as /tools/assistant.

Conversation Webhooks

ElevenLabs sends webhooks to Convex to track the conversation lifecycle:

• POST /webhooks/elevenlabs/conversation-start: Called when a conversation begins. This is used to map the elevenlabsConversationId to the userId passed from Twilio.
• POST /webhooks/elevenlabs/post-call: Called after the conversation ends. It delivers a payload containing the full transcript, summary, call metadata, and other analytics, which are then stored in the Convex database.

Sources: convex/http.ts:417-598

AI Logic and Salesforce Interaction

The core intelligence of the system resides in the askSalesforce Convex action, which is triggered by the /tools/assistant webhook.

Sources: convex/ai.ts, convex/http.ts:114-177

This diagram shows the logic flow within the askSalesforce action.

1. Intent Interpretation with Claude

The askSalesforce action uses Anthropic's Claude model to interpret the user's request.

1. Context Building: It constructs a detailed system prompt that includes the standard Salesforce schema, available tools, and, crucially, any custom objects and fields specific to the user's Salesforce organization. This custom schema is fetched from the orgMetadata table.
2. API Call: It calls the interpretWithClaude function, which sends the conversation history and the system prompt to the Claude API.
3. Structured Response: Claude is prompted to return a structured JSON object containing the action to perform (e.g., search, create, update), the target objectType, and any necessary parameters like searchTerm, soql query, or fields to update.

Sources: convex/ai.ts:100-192, convex/ai.ts:308-386

Here is a snippet of the system prompt that instructs Claude on how to structure its response.

Based on the user's message, determine what Salesforce action to take. Respond with a JSON object:

{
  "action": "search" | "query" | "create" | "update" | "log_call" | "clarify",
  "objectType": "Account" | "Contact" | "Lead" | "Opportunity" | "Task" | "Case" (if applicable),
  "recordId": "18-char Salesforce ID" (if updating a specific record),
  "searchTerm": "what to search for" (if searching),
  "soql": "SELECT ... FROM ..." (if running a specific query),
  "fields": { "FieldName": "value" } (if creating/updating),
  "clarificationQuestion": "question to ask user" (if action is clarify),
  "response": "what to say to the user (keep it SHORT for voice)",
  "followUp": "a helpful next action suggestion (optional, keep SHORT)"
}

Sources: convex/ai.ts:311-324

2. Action Execution

Based on the action field in Claude's response, a switch statement in askSalesforce routes the request to the appropriate Convex action that wraps the Salesforce API. For example, if the action is create, it calls api.salesforce.createRecord with the provided fields.

Sources: convex/ai.ts:195-305

3. Response Formatting

After the Salesforce action is complete, the result is passed to a formatting function (e.g., formatSearchResponse, formatOpportunities) to generate a concise, human-readable string. This string is then sent back to the ElevenLabs agent to be spoken to the user.

Sources: convex/ai.ts:392-544

Summary

The voice interaction system in TalkCRM is a sophisticated integration of Twilio, ElevenLabs, and a Convex backend. It uses Twilio for phone connectivity, authenticates users via caller ID, and connects them to an intelligent ElevenLabs agent. This agent leverages the Claude AI model via Convex actions to understand natural language, execute complex Salesforce operations, and provide a seamless, hands-free CRM experience for sales reps on the go.

Sources: README.md, PROJECT_DESCRIPTION.md