Curiosity Engine

Relevant source files

The following files were used as context for generating this wiki page:

• monGARS/core/cortex/curiosity_engine.py
• monGARS/core/cortex/prompt_enricher.py
• tests/test_curiosity_engine.py
• README.md
• scripts/generate_monGARS_llm_dataset.py
• configs/agents/agents_config.json

The Curiosity Engine is a core component of the monGARS cognition system, designed to proactively detect knowledge gaps in a conversation and trigger external research to enrich the context provided to the Large Language Model (LLM). It is a key part of the conversational pipeline, augmenting user queries with memory, curiosity-driven research, and neuro-symbolic hints before they reach the LLM. This process ensures that the assistant's responses are more informed, accurate, and contextually aware.

The engine operates by analyzing the user's latest query against conversation history and a knowledge graph. If it identifies a potential "knowledge gap," it formulates a research query and uses a dedicated workflow to fetch, verify, and summarize external information, which is then supplied as additional context for the final response generation.

Sources: monGARS/core/cortex/curiosity_engine.py:204-206, README.md:21-23, README.md:92-95, scripts/generate_monGARS_llm_dataset.py:382-383

Architecture and Core Components

The Curiosity Engine is primarily implemented in the CuriosityEngine class. It collaborates with several other services and libraries to perform its functions, including optional dependencies for advanced features like knowledge graph lookups and NLP.

Sources: monGARS/core/cortex/curiosity_engine.py:204-206

Key Classes and Collaborators

Component	Class/Library	Responsibility	Source File
Curiosity Engine	CuriosityEngine	Orchestrates gap detection and research.	curiosity_engine.py
Research Workflow	ResearchWorkflow	Manages the process of fetching and summarizing external information.	curiosity_engine.py
Web Scraper	Iris	Fetches and parses content from web documents.	curiosity_engine.py
Search Orchestrator	SearchOrchestrator	Executes web searches across configured providers.	curiosity_engine.py
Content Verifier	Verifier	Cross-checks search results to determine confidence and find consensus.	curiosity_engine.py
LLM Integration	LLMIntegration	Provides text embedding capabilities for vector similarity search.	curiosity_engine.py
NLP Model	spacy	Extracts named entities from queries. Falls back to a dummy implementation if the model is not found.	curiosity_engine.py
Knowledge Graph	AsyncGraphDatabase	Optional Neo4j driver for checking entity existence. Falls back to a no-op driver if not configured.	curiosity_engine.py

This diagram shows the relationships between the CuriosityEngine and its main collaborators.

Knowledge Gap Detection

The primary function of the engine is detect_gaps, which determines if the system has enough information to answer a query or if it needs to perform research.

The process follows these steps:

1. It first checks if the user's last query is similar to previous queries in the conversation history using vector similarity. If the number of similar historical queries meets the curiosity_minimum_similar_history threshold, it assumes knowledge is sufficient.
2. If not, it extracts named entities from the query using spaCy.
3. It then checks for the presence of these entities in a knowledge graph (if configured).
4. If the number of missing entities is greater than or equal to the curiosity_graph_gap_cutoff, a "knowledge gap" is declared.
5. A new research query is formulated from the original query and the missing entities.
6. The ResearchWorkflow is invoked to perform external research and return additional context.

Sources: monGARS/core/cortex/curiosity_engine.py:441-479

This flowchart illustrates the logic of the detect_gaps method.

Vector Similarity Search

To avoid redundant research, the engine compares the current query to recent historical queries.

• History Loading: It loads recent queries from the database (ConversationHistory table) and combines them with any fallback history provided in the context.
• Embedding vs. Token Fallback: If an embedding model is available via LLMIntegration, it calculates the cosine similarity between the query vector and history vectors. If embeddings are unavailable or fail, it falls back to a token-based Jaccard similarity.
• Threshold: A query is considered "similar" if the similarity score is above the curiosity_similarity_threshold.

Sources: monGARS/core/cortex/curiosity_engine.py:481-605, tests/test_curiosity_engine.py:12-45

Knowledge Graph (KG) Integration

The KG is used to quickly verify if the system "knows" about the entities mentioned in a query.

• Optional Driver: The engine connects to a Neo4j database if NEO4J_URI, NEO4J_USER, and NEO4J_PASSWORD environment variables are set. Otherwise, it uses a no-op driver that always reports entities as missing.
• Batched Lookups: To improve performance, entity lookups are batched into a single UNWIND query.
• Caching and In-flight Request Handling: The results of KG lookups are cached in a TTLCache (_kg_cache) to reduce database load. The engine also coalesces concurrent requests for the same entity into a single in-flight query, preventing redundant lookups.

Sources: monGARS/core/cortex/curiosity_engine.py:136-163, 737-800

This sequence diagram illustrates the batched KG lookup process, including caching and in-flight request handling.

External Research and Context Enrichment

When a knowledge gap is confirmed, the ResearchWorkflow class is responsible for performing external research. It also powers the build_prompt_context method, which enriches prompts with verified information.

The workflow is as follows:

1. Cache Check: It first checks a local TTLCache (_research_cache) for a previously fetched response to the same query.
2. Search and Verify: If not cached, it uses SearchOrchestrator to get web search results and Verifier to cross-check them, producing a VerifiedBundle.
3. Summarization: It concurrently attempts to fetch summaries from two sources:
   • The primary document retrieval service, configured via DOC_RETRIEVAL_URL.
   • The Iris service, which fetches and parses the primary citation from the search results.
4. Fallback: If both summarization methods fail, it uses Iris.search() as a final fallback.
5. Response Rendering: The final output is a formatted string containing the combined summary, agreed-upon facts, noted disagreements, source citations, and a confidence score.

Sources: monGARS/core/cortex/curiosity_engine.py:1024-1200

This flowchart shows the ResearchWorkflow execution path.

The final enriched context is built using build_context_snippet from prompt_enricher.py, which formats the document and bundle information into a structured snippet for the LLM.

Sources: monGARS/core/cortex/curiosity_engine.py:438, monGARS/core/cortex/prompt_enricher.py

Configuration

The Curiosity Engine's behavior is controlled by several settings, typically loaded via monGARS.config.get_settings() and environment variables.

Setting	Environment Variable	Description	Default	Source File
similarity_threshold	CURIOSITY_SIMILARITY_THRESHOLD	Cosine/Jaccard similarity score required to consider history items similar.	0.8	curiosity_engine.py:221
similar_history_threshold	CURIOSITY_MINIMUM_SIMILAR_HISTORY	Number of similar history items required to skip research.	2	curiosity_engine.py:224
graph_gap_cutoff	CURIOSITY_GRAPH_GAP_CUTOFF	Number of missing entities required to trigger research.	2	curiosity_engine.py:227
_kg_cache_ttl	CURIOSITY_KG_CACHE_TTL	TTL for the knowledge graph entity cache in seconds.	300	curiosity_engine.py:241
_kg_cache_max_entries	CURIOSITY_KG_CACHE_MAX_ENTRIES	Max entries in the KG cache.	512	curiosity_engine.py:244
_research_cache_ttl	CURIOSITY_RESEARCH_CACHE_TTL	TTL for the external research results cache in seconds.	900	curiosity_engine.py:253
_research_cache_max_entries	CURIOSITY_RESEARCH_CACHE_MAX_ENTRIES	Max entries in the research cache.	256	curiosity_engine.py:256
Knowledge Graph URI	NEO4J_URI	Connection URI for the Neo4j database.	None	curiosity_engine.py:143
Knowledge Graph User	NEO4J_USER	Username for the Neo4j database.	None	curiosity_engine.py:144
Knowledge Graph Password	NEO4J_PASSWORD	Password for the Neo4j database.	None	curiosity_engine.py:145
Document Retrieval URL	DOCUMENT_RETRIEVAL_URL	Endpoint for the external document retrieval service.	None	curiosity_engine.py:1124

Sources: monGARS/core/cortex/curiosity_engine.py

Telemetry

The engine is instrumented with OpenTelemetry counters to provide visibility into its operations.

Metric Name	Description	Source File
curiosity_external_research_requests	Number of external research requests initiated, with a channel attribute (iris or document_service).	curiosity_engine.py:59-64
curiosity_kg_lookup_events	Knowledge graph lookup hits and misses, with an outcome attribute (hit or miss).	curiosity_engine.py:65-69
curiosity_research_cache_events	Cache events for external research queries, with an event attribute (hit or miss).	curiosity_engine.py:70-74

Sources: monGARS/core/cortex/curiosity_engine.py:59-74

Summary

The Curiosity Engine is a sophisticated system that enhances the monGARS assistant's cognitive abilities. By intelligently deciding when to seek external information, it balances performance with response quality. Its multi-layered approach—combining historical context, a knowledge graph, and a robust external research workflow—allows it to fill its own knowledge gaps, leading to more comprehensive and reliable interactions. The engine's use of caching, asynchronous operations, and optional dependencies makes it both efficient and resilient.