Gemini Voice Assistant
Relevant source files
The following files were used as context for generating this wiki page:
The Gemini Voice Assistant is a hands-free co-pilot designed for campervan drivers using the P4N trip planner. It enables users to interact with their active trip (A → B → C) through spoken commands, allowing them to search for overnight spots, find amenities (like fuel or supermarkets), navigate to specific results, and adjust trip settings without manual input.
The system utilizes the Web Speech API for transcription and text-to-speech, Google's Gemini Pro Flash model for natural language understanding and tool-selection, and a deterministic backend to execute searches against real data. Crucially, the AI model never invents location data; it only interprets intent to trigger specific application logic.
Sources: src/lib/gemini.ts:1-10, src/app/api/assistant/route.ts:40-45, src/components/VoiceButton.tsx:1-10
System Architecture and Data Flow
The architecture follows a request-response pattern where the client captures audio, converts it to text, and sends it to a specialized assistant API route. The API coordinates between the Gemini AI model and the project's internal search and planning libraries.
High-Level Flow
- Capture: The
VoiceButtoncomponent usesSpeechRecognitionto transcribe user speech. - Request: The transcript is sent to
/api/assistantalong with current trip context (polyline, progress, settings). - Inference: The Gemini model (
decidefunction) analyzes the text and context to output a structured JSON "decision." - Execution: The backend executes the chosen action (e.g.,
gatherPlaces,amenitySearch). - Response: The system returns a spoken reply string and a transformed data payload (ranked spots or POIs) to the UI.
The following sequence diagram illustrates the voice command lifecycle:
Gemini AI Integration
The system uses google-generative-ai to interface with the Gemini model. It employs a "Tool Choice" pattern where the AI acts as an intent classifier that maps natural language to a predefined schema.
Decision Schema
The model is constrained to return an object following the DECISION_SCHEMA. Key properties include:
- action: The primary intent (e.g.,
search_p4n,search_amenity,navigate). - amenityKind: Specified if searching for OSM amenities (fuel, toilets, etc.).
- windowMinutes / windowKm: Used for "Look Ahead" queries (e.g., "in the next 45 minutes").
- speech: A short, spoken-optimized sentence generated by the AI to explain what it is doing.
Intent Mapping
| Spoken Phrase Example | Action Chosen | Argument Details |
|---|---|---|
| "Find a spot with a dog park nearby" | search_p4n | nearDogPark: true |
| "I need petrol in the next 50 km" | search_amenity | amenityKind: "fuel", windowKm: 50 |
| "Navigate to number two" | navigate | optionNumber: 2 |
| "Make the corridor wider" | adjust_corridor | maxKmFromRoad (calculated value) |
Sources: src/lib/gemini.ts:16-65, src/app/api/assistant/route.ts:55-75
Backend Execution Logic
Once Gemini determines the intent, the /api/assistant route executes the actual search logic. This ensures that the AI's tendency to "hallucinate" is neutralized by only allowing it to trigger verified data lookups.
Search Windows (Look Ahead)
If the user specifies a time or distance (e.g., "up ahead"), the assistant resolves a ProgressWindow. This window represents a slice of the route polyline based on the driver's currentProgressKm.
- Time to Distance: If the user asks for minutes, the system assumes an average speed (e.g., 80 km/h) to convert time into a distance band along the route.
- Filtering: The search results are then passed through
withinWindowto ensure only spots in that specific segment are returned.
Deterministic Ranking
Results are ranked using the score function in src/lib/planner.ts. The assistant prioritizes:
- Detour Distance: Proximity to the planned road polyline (
kmFromRoad). - User Preferences: Bonuses for dog park proximity or required services (water, electric).
- Ratings: Weighted star ratings from Park4Night.
Sources: src/app/api/assistant/route.ts:57-74, src/lib/planner.ts:125-145
Voice Components and Web API Usage
The frontend interaction is managed by two primary entities in src/components/VoiceButton.tsx:
Speech Recognition
The VoiceButton component wraps the webkitSpeechRecognition (or standard SpeechRecognition) interface. It is configured for:
- Language:
en-GB. - Interim Results: Disabled (only full transcripts are processed).
- Continuous: Disabled (listens for a single command, then stops).
Speech Synthesis
The speak() utility uses the window.speechSynthesis API to provide audio feedback. It cancels any ongoing speech before starting a new SpeechSynthesisUtterance to ensure the driver is not overwhelmed by overlapping audio.
export function speak(text: string) {
if (typeof window === "undefined" || !("speechSynthesis" in window)) return;
try {
window.speechSynthesis.cancel();
const u = new SpeechSynthesisUtterance(text);
u.lang = "en-GB";
u.rate = 1.05;
window.speechSynthesis.speak(u);
} catch { /* ignore */ }
}
Sources: src/components/VoiceButton.tsx:35-55, src/components/VoiceButton.tsx:103-115
Summary
The Gemini Voice Assistant serves as a critical safety and convenience feature for the P4N improved trip planner, enabling hands-free operation during driving. By combining the natural language capabilities of Gemini with deterministic search tools, the system provides a reliable "co-pilot" experience that accurately surfaces relevant travel data based on real-time trip progress.
The implementation successfully separates the AI-driven intent interpretation from the data-driven result generation, ensuring both conversational flexibility and technical accuracy.