Investigation Pipeline
Relevant source files
The following files were used as context for generating this wiki page:
The Investigation Pipeline is a local research orchestrator that plans sub-queries, executes multi-provider searches, and extracts verifiable evidence into a structured EvidencePack. It automates the process of gathering source material from the web and repositories to answer complex investigative questions.
The pipeline operates by fanning out searches across multiple providers, merging results through fusion ranking, and reading the most relevant sources. It extracts deterministic passages based on key terms and can optionally generate a summary brief or perform claim-corroboration to verify statements against the gathered evidence.
Architecture and Data Flow
The pipeline is implemented as a thin data-returning composition of existing search and read capabilities. It coordinates several distinct modules to transform a user question into an evidence-rich data structure.
High-Level Flow
- Planning: Generates sub-queries from the input question or a context file.
- Search Fan-out: Executes the sub-query grid across one or more search arms.
- Source Selection: Identifies distinct source URLs from merged and clustered search results.
- Parallel Reading: Fetches content from sources using a bounded-concurrency pool of reader suppliers.
- Evidence Extraction: Identifies verbatim passage quotes within the source content that match derived key terms.
- Synthesis (Optional): Generates a brief summary using Z.AI chat capabilities.
- Verification (Optional): Matches sentence-claims from a statement against the extracted passages to determine corroboration or contradiction.
Sources: packages/scoutline/src/commands/investigate.ts:35-75, packages/scoutline/src/index.ts:2519-2536
System Orchestration Diagram
The following diagram illustrates the flow of data through the pipeline components:
The diagram shows the sequential transformation of a question into an EvidencePack, including the optional synthesis step.
Sources: packages/scoutline/src/commands/investigate.ts:35-75, packages/scoutline/src/index.ts:2543-2550
Sub-query Planning
The planner module determines the search strategy by deriving a set of specific queries from the user's input. It uses three tiers of precedence:
- Pipes: If the question contains an unescaped
|, the planner splits the input into explicit sub-queries (max 8). - Context File: If a
--contextpath is provided and no pipes are present, the planner derives up to 8 sub-queries from headings and questions found in the local file. - Template: If no pipes or context are provided, the planner applies deterministic transforms to the bare question to generate up to 5 sub-queries, including key terms and perspectives like "evidence" or "criticism".
Sources: packages/scoutline/src/lib/investigate-planner.ts:22-60, packages/scoutline/src/commands/investigate.ts:153-176
Planner Precedence Table
| Tier | Trigger | Max Sub-queries | Description |
|---|---|---|---|
| Pipes | ` | ` in question | 8 |
| Context | --context <path> | 8 | Derived from headings/questions in a notes file. |
| Template | Default | 5 | Automatic deterministic transforms of the question. |
Sources: packages/scoutline/src/commands/investigate.ts:153-176, packages/scoutline/src/lib/investigate-planner.ts:161-180
Evidence Extraction and Assembly
Once sources are read, the pipeline extracts evidence to populate the EvidencePack. This process is deterministic and focuses on verbatim transparency.
Passage Extraction
The extraction logic identifies passages containing derived key terms. It splits the source content into sentences and identifies those with the highest term density. Each passage in the result includes a quote (the verbatim text) and a charRange (start and end indices relative to the source content).
Sources: packages/scoutline/src/lib/investigate-extract.ts:10-40, packages/scoutline/src/commands/investigate.ts:743-764
EvidencePack Structure
The EvidencePack is the final data schema (version 1) produced by the pipeline.
| Field | Type | Description |
|---|---|---|
schemaVersion | number | Always 1. |
question | string | The original input question. |
subQueries | string[] | The list of queries actually searched. |
sources | EvidenceSource[] | Array of successful reads with metadata and passages. |
coverage | object | Metrics including search hits, arms used, and unread sources. |
brief | string (optional) | The Z.AI-generated summary brief. |
verify | object (optional) | Claim verification verdicts and evidence pointers. |
Sources: packages/scoutline/src/capabilities/investigation.ts:18-40, packages/scoutline/src/commands/investigate.ts:766-785
Verification Mode
The --verify flag engages a specialized mode for statement corroboration. In this mode, the input positional argument is treated as a statement of fact rather than a question.
Claim Processing
- Splitting: The statement is split into individual sentence-claims (max 8).
- Searching: Each claim is searched verbatim as a sub-query.
- Verdict Determination:
- Corroborated: At least one matching passage is found with zero negation cues.
- Contradicted: At least one matching passage is found that contains a negation cue (e.g., "never", "not", "false").
- Unresolved: No matching passages are found in the gathered evidence.
Sources: packages/scoutline/src/lib/investigate-claims.ts:14-45, packages/scoutline/src/commands/investigate.ts:184-210
Verification Logic Diagram
The sequence shows how claims are extracted from a statement and subsequently matched against evidence to produce verdicts.
Sources: packages/scoutline/src/lib/investigate-claims.ts:114-160, packages/scoutline/src/commands/investigate.ts:782-785
Output Budgeting
The pipeline supports a --max-chars flag to bound the size of the generated EvidencePack. It applies a specific ladder of reduction rules to fit the budget while preserving high-value information.
Investigation Ladder Rules
- Trim Passages: Truncates passage quotes from the end and adjusts character ranges. This is the first rule attempted.
- Drop Late Sources: Removes the last source in the array along with its associated evidence pointers.
- Drop Evidence Pointers (Verify Mode): For verification packs, it drops all evidence pointers for claims while keeping the claim text and verdicts.
Sources: packages/scoutline/src/commands/investigate.ts:310-365
Synthesis Escape Hatch
The --synthesize flag attaches a human-readable brief to the data pack. This is an additive-only step that occurs after the EvidencePack is fully assembled. It uses a Z.AI chat completion with a system prompt instructing the model to use only the quoted evidence and avoid inventing facts.
Sources: packages/scoutline/src/commands/investigate.ts:710-740, packages/scoutline/src/commands/investigate.ts:787-814
Synthesis Prompt Components
- Question: The original user question.
- Sub-queries: The planned grid searched by the pipeline.
- Quotes: A list of extracted passage quotes, capped at 20 to fit context windows.
Sources: packages/scoutline/src/commands/investigate.ts:115-135
Conclusion
The Investigation Pipeline provides a robust framework for automated web and repository research. By combining deterministic sub-query planning, fusion-merged search results, and term-based passage extraction, it creates a high-fidelity evidence base for agents and developers. Its support for claim verification and Z.AI-driven synthesis makes it a comprehensive tool for validating information and summarizing findings directly from source material.