Test Scenarios

Relevant source files

The following files were used as context for generating this wiki page:

• scenarios.py
• README.md
• app.py
• orchestrator.py
• memory_utils.py

The BrightPath AI Operations system utilizes a suite of 18 pre-built test scenarios to validate the multi-agent pipeline's ability to process inbound sales and customer service communications. These scenarios serve as standardized inputs for the Triage, Specialist (Sales/CS), and Manager Briefing agents, ensuring that classification, routing, and response drafting logic remain accurate across various customer interactions.

Test scenarios are categorized into two primary groups: Original Scenarios, which focus on baseline classification and single-turn responses, and Memory-Dependent Scenarios, which are specifically designed to test the system's ability to maintain state and carry context across multiple interactions using the project's file-based memory system.

Sources: scenarios.py:8-13, README.md:10-20, README.md:92-115

Scenario Architecture and Data Flow

Test scenarios are stored in a central repository within scenarios.py and are imported by both the Streamlit web UI (app.py) and the terminal-based orchestrator.py. When a scenario is selected, the message string is injected into the processing pipeline.

Pipeline Execution Flow

The following diagram illustrates how a scenario message moves through the system:

Scenario Classifications

Scenarios are defined in a dictionary structure where the key serves as a descriptive label and the value is the inbound message content.

Original Scenarios (1-10)

These scenarios cover fundamental business cases including new leads, complaints, and administrative requests.

ID	Scenario Title	Primary Objective
1	New sales lead	Tests lead qualification and corporate package identification.
2	Customer booking change	Validates policy application for workshop transfers.
3	Angry customer complaint	Tests sentiment detection and empathy in drafting.
4	VIP upsell	Validates recognition of "Annual Partners" (e.g., Nexus Digital).
9	Refund — policy edge case	Tests logic for redundancy-based refund flexibility.
10	Manager daily briefing	Triggers the administrative reporting path using DB context.

Sources: scenarios.py:16-59, app.py:289-300

Memory-Dependent Scenarios (11-18)

These scenarios require the system to reference customer_history.md, sales_notes.md, and open_tasks.md to respond correctly.

ID	Scenario Title	Memory Interaction
11	Repeat complaint escalation	Checks history for prior complaints; triggers manager alert if count >= 3.
13	Overdue invoice flagged	Cross-references unpaid invoices in CRM while discussing new workshops.
16	Duplicate lead (same company)	Detects if different contacts from the same firm are in the pipeline.
18	Slow payer booking attempt	References payment history to hold new bookings until debt is settled.

Sources: scenarios.py:65-103, memory_utils.py:108-144, memory_utils.py:202-221

Execution Logic and Memory Persistence

The system implements a specific post-execution hook for scenarios 11-18. While original scenarios typically result in database logging, memory-dependent scenarios trigger write-backs to Markdown files via memory_utils.py.

Sequence of State Changes

Implementation Details

The SCENARIOS dictionary and SCENARIO_GROUPS metadata allow for dynamic UI rendering.

# scenarios.py:108-111
SCENARIO_GROUPS = {
    "Original Scenarios": [str(i) for i in range(1, 11)],
    "Memory-Dependent Scenarios": [str(i) for i in range(11, 19)],
}

In app.py, the run_pipeline function utilizes the scenario_key to determine if a memory update is required after the agents have completed their tasks.

# app.py:328-330
memory_updated = run_memory_update(scenario_key, triage, agent_result)
results["memory_updated"] = memory_updated

Sources: scenarios.py:108-111, app.py:328-330, memory_utils.py:248-262

Conclusion

Test Scenarios are the primary mechanism for validating the BrightPath AI pipeline. By categorizing tests into baseline classification (1-10) and persistent state management (11-18), the project ensures that agents can handle both immediate requests and long-term customer relationships. The integration between scenarios.py and memory_utils.py creates a closed-loop system where testing outcomes are recorded as the new baseline for subsequent system runs.