Local Model Integration (Phi-3)

Relevant source files

The following files were used as context for generating this wiki page:

• backend/modules/ml/phi3_model.py
• backend/ml_config/model_config.py
• backend/services/ai_service.py
• Ai-Assistant-v2/auth-service/server.js
• Ai-Assistant-v2/auth-service/routes/ai.js
• README.md

Introduction

The Local Model Integration (Phi-3) feature enables the AI Assistant to provide privacy-focused, offline-capable, and fast conversational responses without relying on external APIs. This implementation utilizes Microsoft's Phi-3-mini-4k-instruct model in GGUF format, integrated via the llama-cpp-python library. Sources: README.md:1-5, backend/modules/ml/phi3_model.py:1-10

Within the 3-tier architecture, the Phi-3 model serves as a core component of the Python AI Service. It provides an alternative to the Google Gemini API, allowing users to switch models based on their needs for privacy or advanced reasoning. Sources: README.md:10-50, Ai-Assistant-v2/auth-service/routes/ai.js:150-180

Architecture and Data Flow

The integration follows a tiered flow where requests originate in the React frontend, pass through the Node.js Auth/Orchestration service for context enrichment, and are finally processed by the Python AI Service hosting the Phi-3 model. Sources: README.md:10-45, Ai-Assistant-v2/auth-service/server.js:250-280

System Orchestration

The following diagram illustrates the sequence of operations when a user selects the Phi-3 model for a chat message.

Model Configuration and Initialization

The Phi-3 model is managed by the Phi3Model wrapper class. It is configured to run on the CPU to ensure stability across different hardware environments. Sources: backend/ml_config/model_config.py:10-25, backend/modules/ml/phi3_model.py:15-40

Core Configuration Parameters

The model settings are defined in model_config.py to balance performance and response quality.

Parameter	Value	Description
model_path	models/phi3/...	Path to the .gguf model file.
n_ctx	2048	Context window size for conversation history.
n_threads	8	Number of CPU threads dedicated to inference.
temperature	0.3	Controls randomness (lower is more deterministic).
use_gpu	False	Explicitly disabled due to DirectML instability.
repeat_penalty	1.1	Prevents the model from repeating the same phrases.

Sources: backend/ml_config/model_config.py:10-25, backend/modules/ml/phi3_model.py:20-30

Prompt Engineering and Context Management

The system utilizes the ChatML format to structure interactions for the Phi-3 model. This involves wrapping system instructions, previous conversation turns, and the current user query in specific delimiters. Sources: backend/modules/ml/phi3_model.py:100-130

ChatML Formatting Logic

The _build_prompt method constructs the final string sent to the model:

1. System Prompt: Injected at the start (e.g., "You are a friendly AI assistant"). Sources: backend/ml_config/model_config.py:55-65
2. Context: Includes recent messages and relevant memories retrieved via vector search. Sources: backend/services/ai_service.py:30-45
3. Delimiters: Uses <|system|>, <|user|>, and <|assistant|> tags. Sources: backend/modules/ml/phi3_model.py:110-125

Technical Implementation Details

The Phi3Model Class

Located in backend/modules/ml/phi3_model.py, this class handles the lifecycle of the model:

• __init__: Triggers _load_model to instantiate the Llama object from llama-cpp-python.
• generate: The primary method for inference, accepting prompts and context.
• unload: Cleans up memory by deleting the model instance.

Sources: backend/modules/ml/phi3_model.py:15-55, backend/modules/ml/phi3_model.py:175-185

AI Service Orchestration

The AIService in backend/services/ai_service.py acts as a stateless coordinator. It receives requests from the Node.js backend, formats the context, and routes the call to either Phi3Model or GeminiModel. Sources: backend/services/ai_service.py:10-50

# Snippet from AIService.generate illustrating model routing
if model == 'gemini':
    response = self.gemini_model.generate(prompt=user_message, system_prompt=system_prompt, context=context)
else:  # Default to phi3
    response = self.phi3_model.generate(prompt=user_message, system_prompt=system_prompt, context=context)

Sources: backend/services/ai_service.py:48-55

Conclusion

The Phi-3 integration provides a robust local alternative to cloud-based AI. By utilizing the GGUF format and optimized CPU inference settings, the system ensures that users can maintain productive conversations with the AI assistant even in offline environments or scenarios requiring strict data privacy. The use of ChatML formatting and a dedicated wrapper class allows for seamless switching between local and cloud models. Sources: README.md:20-35, backend/modules/ml/phi3_model.py:1-10