Dual AI Model System

Relevant source files

The following files were used as context for generating this wiki page:

• backend/modules/ml/phi3_model.py
• backend/modules/ml/gemini_model.py
• backend/services/ai_service.py
• backend/ml_config/model_config.py
• Ai-Assistant-v2/auth-service/services/aiService.js
• backend/app.py
• Ai-Assistant-v2/auth-service/routes/ai.js

The Dual AI Model System is a core architectural feature of the AI Assistant platform that provides users with a choice between two distinct large language models (LLMs): a local Phi-3 model and a cloud-based Google Gemini model. This system allows the application to balance between offline capability, privacy, and speed (via Phi-3) and advanced reasoning or internet-connected knowledge (via Gemini).

The system is orchestrated through a tiered architecture where a Node.js Auth & Orchestration service manages user state and conversation history, while a Python-based AI service handles the actual model inference and logic. Users can switch models seamlessly per conversation through the frontend interface or via specialized settings endpoints.

Sources: README.md, backend/services/ai_service.py:11-18, Ai-Assistant-v2/auth-service/routes/ai.js:154-184

System Architecture and Orchestration

The system follows a 3-tier architecture. The Node.js Orchestration service acts as the primary gateway, handling JWT authentication and database retrieval (MongoDB) before forwarding processed requests to the Python AI Service.

Data Flow for Chat Requests

When a user sends a message, the system orchestrates several steps to provide a context-aware response:

1. Context Loading: The Node.js service retrieves the last 10 messages of the conversation.
2. Memory Retrieval: Vector-based similarity search identifies relevant user memories.
3. Inference Request: A payload containing the prompt, context, memories, and selected model ID is sent to the Python service.
4. Model Execution: The Python AIService selects either Phi3Model or GeminiModel based on the request.
5. Metadata Generation: The system simultaneously detects user intent and sentiment.

Model Specifications

Phi-3 Mini (Local)

The Phi-3 model is implemented using the llama-cpp-python library, utilizing a GGUF quantized format. It is configured to run primarily on the CPU to ensure stability across different hardware environments without requiring specific GPU drivers.

Parameter	Configuration Value
Model Path	models/phi3/Phi-3-mini-4k-instruct-q4.gguf
Context Window (n_ctx)	2048 tokens
Temperature	0.3
Max Output Tokens	500
Execution Hardware	CPU (8 threads)

Sources: backend/ml_config/model_config.py:11-23, backend/modules/ml/phi3_model.py

Google Gemini (Cloud)

The Gemini integration uses the google-generativeai SDK. It provides advanced capabilities and utilizes a different prompt formatting logic compared to the local model.

Parameter	Configuration Value
Model Name	gemini-flash-latest
Temperature	0.7
Top P	0.95
Max Output Tokens	500
Connection	Cloud API

Sources: backend/ml_config/model_config.py:27-33, backend/modules/ml/gemini_model.py:29-37

Model Selection and Switching

The system persists the model choice at two levels: within individual Conversation documents and in the User settings.

Configuration Hierarchy

1. User Default: Set in the user profile; determines the model for new conversations.
2. Conversation Active Model: Stored in the conversation document; determines the model used for subsequent messages in that thread.

API Endpoints for Model Management

Endpoint	Method	Description
/api/v1/ai/model	GET	Retrieves the current model for the user's most recent conversation.
/api/v1/ai/model	POST	Updates the active model ID (phi3 or gemini) for the user.
/api/v1/settings/model	POST	Persists a default model selection to the user's account settings.

Sources: Ai-Assistant-v2/auth-service/routes/ai.js:132-184, Ai-Assistant-v2/auth-service/server.js:158-180

Implementation Details: AIService

The AIService class in the Python backend serves as the primary router. It is responsible for injecting relevant memories into the system prompt before passing the data to the specific model wrapper.

# Simplified Logic from backend/services/ai_service.py
def generate(self, messages, memories=None, model='phi3'):
    system_prompt = SYSTEM_PROMPTS.get('chat')
    
    # Inject retrieved memories into the context
    if memories:
        memory_text = "\n".join([f"- {m['fact']}" for m in memories])
        system_prompt += f"\n\nRelevant Memories:\n{memory_text}\n"
    
    # Route to selected model
    if model == 'gemini':
        response = self.gemini_model.generate(...)
    else:
        response = self.phi3_model.generate(...)
        
    return {
        'response': response,
        'intent': self.ml_service.detect_intent(...),
        'sentiment': self.ml_service.analyze_sentiment(...)
    }

Sources: backend/services/ai_service.py:20-63

Summary

The Dual AI Model System provides a robust framework for hybrid AI interaction. By decoupling the orchestration logic in Node.js from the inference logic in Python, the project enables flexible model swapping and context enrichment via memories, ensuring that the assistant remains responsive and personalized regardless of whether the local Phi-3 or cloud Gemini model is active.