Perception & Social Modules
Relevant source files
The following files were used as context for generating this wiki page:
The Perception and Social modules provide CORTEX with environmental awareness and human-centric interaction capabilities. The Perception module captures and analyzes external environmental states, such as screen activity on macOS, while the Social module manages a weighted graph of human relationships, communication preferences, and empathic user modeling. These modules transform raw external signals into memory nodes and structured state assessments that influence agent behavior.
Sources: README.md:120-134, src/mcp/server.ts:717-783
Perception and Environmental Awareness
The Perception module utilizes the cortex_observe tool to capture the current state of the user's computing environment. This system detects the active application, window titles, and visible content. Observations are stored in the database as memory_nodes with a specific source_type of observation. This allows the agent to maintain a temporal record of what the user is working on, which categorized as environmental context in the memory graph.
Sources: src/mcp/server.ts:717-731, src/api/graph.ts:18-20
Observation Life Cycle
Environmental observations are treated as ephemeral context. During the Slow-Wave Sleep (SWS) phase of the Dream Cycle, the system executes "Tier 4" pruning. This process deletes all screen observations older than seven days to prevent the memory store from becoming cluttered with stale environmental data.
Sources: src/dream/dream-cycle.ts:241-249
The diagram shows the flow of perception data from initial capture to its eventual removal during the dream cycle. Sources: src/mcp/server.ts:725-731, src/dream/dream-cycle.ts:241-249
Social Intelligence & Relationship Graph
The Social module manages interactions through the relationship_graph table. It tracks contacts by name and email, assigning a relationship_type (e.g., family, client, partner) and an importance_score. The system maintains a "Personality Model" and "Communication Preferences" for each individual, enabling the agent to tailor its interactions based on historical data.
Sources: src/db/schema.ts:365-388, src/mcp/server.ts:734-750
Key Relationship Features
- Open Items: Tracks pending tasks or topics associated with specific people.
- Contact Freshness: Monitors
last_contactandcontact_frequencyto identify overdue communications. - Interaction History: Links individuals to specific sessions and shared memory nodes.
Sources: src/db/schema.ts:383-386, src/mcp/server.ts:753-763
Relationship Schema Summary
| Field | Type | Description |
|---|---|---|
person_name | varchar | Primary identifier for fuzzy matching |
relationship_type | varchar | Category (client, professional, friend, etc.) |
last_contact | timestamp | ISO date of the most recent interaction |
importance_score | real | Weight used for recall and prioritization |
personality_model | jsonb | Structured data regarding the person's traits |
open_items | jsonb | List of unresolved tasks or follow-ups |
Sources: src/db/schema.ts:365-388
Empathic Modeling (Principal State)
CORTEX maintains a principal_state table to model the human user's internal state. The cortex_assess_state tool analyzes recent message patterns and calendar context to infer the user's energy, stress levels, and focus state (such as hyperfocus or executive dysfunction). These assessments are stored with a confidence_score and are used to adjust the agent's communication style.
Sources: src/db/schema.ts:348-362, src/mcp/server.ts:684-696
The ER diagram illustrates the relationship between the core Agent entity and the modules for social tracking and empathic modeling. Sources: src/db/schema.ts:348-388
User State Indicators
The system tracks specific cognitive and emotional states of the principal:
- Focus State: Classified as
hyperfocus,flow,normal,scattered, orexecutive_dysfunction. - ADHD State: Classified as
managed,restless,overwhelmed, orshutdown. - Valence: Measures the positive or negative emotional charge of the user's recent signals.
Sources: src/db/schema.ts:355-359
Social & Perception Tools (MCP)
The Model Context Protocol (MCP) server exposes several tools for interacting with these modules.
| Tool Name | Action | Parameters |
|---|---|---|
cortex_observe | Captures macOS screen state | store (boolean) |
cortex_relationship | Retrieves contact profile | name (string) |
cortex_assess_state | Infers principal energy/stress | recent_messages (array) |
cortex_bg_thread | Runs relational reasoning | thread_type (relational) |
Sources: src/mcp/server.ts:684-750
Summary
The Perception and Social modules allow CORTEX to function as a context-aware assistant. By observing the environment and modeling the user's emotional and social landscape, the system moves beyond simple data retrieval. The perception data provides immediate context, while the social and empathic models ensure that interactions remain relevant to the user's current mental state and long-term interpersonal goals.
Sources: README.md:120-134, src/mcp/server.ts:705-714