Customizing AI Agents

Relevant source files

The following files were used as context for generating this wiki page:

• agent_configs/SIMPLE-Co-Sales-Agent.json
• agent_configs/Mindfulness-coach.json
• agent_configs/TalkCRM.json
• convex/ai.ts
• convex/http.ts
• README.md
• convex/orgMetadata.ts

In TalkCRM, AI agents are defined and customized through detailed JSON configuration files. These files control every aspect of an agent's behavior, from its personality and voice to the tools it can use and the specifics of its conversational flow. This system allows for the creation of highly specialized agents for various roles, such as a sales agent, a mindfulness coach, or a CRM assistant.

The core of the customization lies in the conversation_config object within the JSON file, which dictates the agent's interaction model, and the backend logic in convex/ai.ts which provides dynamic, org-specific context to the AI, making it aware of custom Salesforce schemas.

Agent Configuration Structure

Each AI agent is defined by a single JSON file. The root of the configuration contains several key objects that structure the agent's definition.

This diagram shows the main components of an agent's configuration file.

• conversation_config: Defines the core conversational experience, including speech recognition, text-to-speech, turn-taking, and the agent's persona and tools.
• platform_settings: Configures how the agent is presented and behaves on the platform, such as the web widget's appearance and privacy settings.
• workflow: Defines advanced, stateful conversation flows using nodes and edges (e.g., start_node).

Sources: agent_configs/SIMPLE-Co-Sales-Agent.json, agent_configs/Mindfulness-coach.json

Core Conversation Configuration (conversation_config)

This section is the heart of agent customization, defining how the agent perceives, thinks, and responds.

Text-to-Speech (TTS)

The tts object controls the agent's voice and speech characteristics, primarily using the ElevenLabs service.

Parameter	Type	Description	Example Value
model_id	string	The ElevenLabs model to use for speech synthesis.	eleven_multilingual_v2
voice_id	string	The specific voice ID from ElevenLabs.	mHV5m7DLaQM0bIAP6BTK
stability	float	Controls the voice's stability. Higher values are more consistent.	0.5
speed	float	The speaking speed of the agent.	1.07
similarity_boost	float	Boosts the similarity to the original voice.	0.75
optimize_streaming_latency	int	Latency optimization level for streaming audio.	3

Sources: agent_configs/SIMPLE-Co-Sales-Agent.json:20-31, agent_configs/Mindfulness-coach.json:20-30

A typical tts configuration looks like this:

"tts": {
    "model_id": "eleven_multilingual_v2",
    "voice_id": "mHV5m7DLaQM0bIAP6BTK",
    "supported_voices": [],
    "suggested_audio_tags": [],
    "agent_output_audio_format": "pcm_16000",
    "optimize_streaming_latency": 3,
    "stability": 0.5,
    "speed": 1.07,
    "similarity_boost": 0.75,
    "text_normalisation_type": "system_prompt",
    "pronunciation_dictionary_locators": []
}

Sources: agent_configs/SIMPLE-Co-Sales-Agent.json:20-31

Agent Persona and Logic (agent)

The agent object defines the agent's identity, initial message, and core intelligence through a system prompt and LLM configuration.

First Message

The first_message is the opening line the agent speaks when a conversation begins. This sets the initial tone and context.

• SIMPLE Co Sales Agent: "Hey - Adam from The Simple Company. What's the biggest headache with your sales process right now?"
• Mindfulness Coach: "Hey there... Thanks for taking this moment for yourself. Let's begin by introducing ourselves. I'm Joe. What's your name?"
• TalkCRM Assistant: "Hey! I'm TalkCRM, your voice assistant for Salesforce. Ready to help you crush it today! I can check your pipeline, find accounts, update deals, or create tasks. What do you need?"

Sources: agent_configs/SIMPLE-Co-Sales-Agent.json:48, agent_configs/Mindfulness-coach.json:47, agent_configs/TalkCRM.json:48

System Prompt (prompt)

The prompt.prompt field is a detailed document that instructs the LLM on how to behave. It is the primary mechanism for customizing an agent's personality, goals, and operational procedures. It typically includes the following sections:

Section	Description
Personality / Mindset	Defines the agent's character, e.g., a "technical founder" or a "gentle, reassuring" coach.
Environment	Describes the context in which the agent operates (e.g., phone calls, WhatsApp).
Tone / Pacing	Specifies the style of speaking, use of language, and conversational flow.
Tools	Lists the external tools the agent can use and when to use them.
Goal	The primary objective of the agent in the conversation (e.g., qualify a lead, guide a meditation).
Guardrails	Strict rules on what the agent should never do (e.g., "Never make up features").
Success Metrics	Defines what a successful conversation looks like for the agent.

Sources: agent_configs/SIMPLE-Co-Sales-Agent.json:54-152, agent_configs/Mindfulness-coach.json:53-128, agent_configs/TalkCRM.json:54-101

LLM Configuration

The prompt object also contains settings for the Large Language Model (LLM) that powers the agent's intelligence.

• llm: The specific model to use (e.g., glm-45-air-fp8, gemini-2.0-flash-001).
• temperature: Controls the randomness of the output. Lower values are more deterministic.
• max_tokens: The maximum number of tokens the model can generate in a single response.

Sources: agent_configs/SIMPLE-Co-Sales-Agent.json:153-156, agent_configs/Mindfulness-coach.json:129-131

Agent Tools

Agents are given capabilities through "tools," which are external functions they can call. In TalkCRM, these are typically api_integration_webhook types that trigger HTTP actions in the Convex backend.

The agent configuration links to tools via tool_ids and defines them in a tools array.

"tools": [
    {
        "type": "api_integration_webhook",
        "name": "calcom_create_booking",
        "description": "Create a new booking for an event. This schedules an appointment.",
        "response_timeout_secs": 20,
        "api_integration_id": "calcom",
        "api_integration_connection_id": "icxn_6101kcppvzk9fv59xn4chmm3mv5v",
        "api_schema_overrides": null
    }
]

Sources: agent_configs/SIMPLE-Co-Sales-Agent.json:181-197

The README.md lists a set of Salesforce-specific tools available to the TalkCRM agent:

Tool	Description
search_salesforce	Find accounts, contacts, opportunities, leads
get_record	Get full details of a specific record
create_record	Create new accounts, contacts, leads, tasks
update_record	Update record fields (stage, status, amount)
log_call	Log call activity to Salesforce
get_my_tasks	View open tasks and to-dos
get_my_pipeline	View opportunities and pipeline summary

Sources: README.md:59-67

The flow of a tool call is illustrated below.

This diagram shows how a user's voice command triggers a series of actions from the ElevenLabs agent to the Convex backend and finally to the Salesforce API.

Sources: README.md:40-46, convex/http.ts:88-154, convex/ai.ts:101-260

Dynamic AI Context with Salesforce Schema

A key aspect of customization is making the AI agent aware of a specific user's Salesforce organization, including custom objects and fields. This is achieved by dynamically injecting a schema context into the AI's system prompt at runtime.

The process is managed by the askSalesforce action in convex/ai.ts and the metadata sync logic in convex/orgMetadata.ts.

This diagram illustrates how the askSalesforce action enriches the AI's prompt with both static standard object definitions and dynamically synced custom object metadata for the specific user's org.

Sources: convex/ai.ts:101-206, convex/orgMetadata.ts:35-64, README.md:73-90

Standard Schema Context

A baseline context of standard Salesforce objects, fields, and rules is hardcoded in the SALESFORCE_SCHEMA_CONTEXT constant. This ensures the AI always understands core CRM concepts like Account, Opportunity, Lead, and Task.

Sources: convex/ai.ts:9-98

Custom Schema Sync

To handle custom objects (e.g., Invoice__c) and fields, the system performs a metadata sync:

1. A weekly cron job (syncAllOrgs) or a manual trigger (syncFromSalesforce) connects to a user's Salesforce org.
2. It fetches all queryable standard and custom objects.
3. For custom objects, it fetches detailed field metadata, including names, types, picklist values, and relationship names.
4. It even queries for a sample record to identify commonly populated fields.
5. This metadata is stored in the orgMetadata table in Convex, linked to the org's instanceUrl.

When a user interacts with the agent, the askSalesforce action retrieves this stored metadata and injects it into the prompt, allowing the AI to construct valid SOQL queries and understand the user's unique data structure.

Sources: convex/orgMetadata.ts:162-386, convex/ai.ts:121-206

Conclusion

Customizing AI agents in TalkCRM is a two-part process. First, developers define an agent's static persona, voice, and capabilities within a JSON configuration file, allowing for diverse agent types like sales assistants and mindfulness coaches. Second, the system dynamically enriches the agent's understanding at runtime by injecting detailed schema information from the specific Salesforce organization it is interacting with. This combination of static configuration and dynamic, data-aware context allows TalkCRM to provide highly tailored and effective AI assistants that can adapt to different business needs and complex, custom CRM environments.