Asynchronous Research
Relevant source files
The following files were used as context for generating this wiki page:
Asynchronous research refers to the system capability of performing multi-step research synthesis and scholarly lookups through long-running operations. The Scoutline CLI manages these tasks by initiating research jobs, tracking their state, and retrieving final reports once processing completes. This system supports keyless scholarly index lookups and deep research synthesis across multiple providers.
Sources: packages/scoutline/tests/science-docs.test.js:68-76, scoutline-augmentation-spec.md:6-15
Architecture and Job Lifecycle
The research system operates as an asynchronous pipeline where a request is first validated and then dispatched to a provider. Because research tasks can take significant time, the system utilizes an async-job-state mechanism to persist job IDs and track progress. Providers like Tavily, Exa, and Parallel implement polling mechanisms to check for job completion.
Research Lifecycle Flow
The following diagram illustrates the transition from an initial user query to the final research report.
The diagram shows how the system creates a job and polls for the result before returning the final report.
Sources: packages/scoutline/tests/controls-conformance.test.js:338-348, packages/scoutline/tests/output-budget-t4.test.js:77-83
Key Components and Data Structures
Research Request Controls
Research operations support specific steering controls that vary by provider. These controls are defined in the ResearchRequest structure.
| Control | Description | Type | Examples |
|---|---|---|---|
model | Specifies the reasoning depth or AI model tier. | String | pro, ultra, deep |
outputLength | Controls the verbosity of the final report. | String | short, long |
citationFormat | Formats how sources are cited in the report. | String | apa, numeric |
domain | Restricts or includes specific domains for research. | String/Array | example.com |
Sources: packages/scoutline/tests/controls-conformance.test.js:1015-1018, packages/scoutline/tests/output-budget-t4.test.js:195-201
Data Models
The final payload returned by an asynchronous research operation follows a schema versioned envelope.
{
"schemaVersion": 1,
"query": "research query",
"model": "auto",
"report": "## Findings\n...",
"sources": [
{ "title": "S1", "url": "https://example.com/s1" }
]
}
Sources: packages/scoutline/tests/output-budget-t4.test.js:195-208
Output Budgeting and Compaction
To ensure research results fit within consumer limits (such as LLM context windows), the research system implements an output budget ladder. The RESEARCH_LADDER prioritizes preserving sources while trimming report bodies.
- Prioritization: Sources and citations survive longest during budget deficits.
- Trimming: The report body is trimmed first, often by bleeding content from later sections before dropping them entirely.
- Omission Markers: Trimmed content is marked with a single omission marker (
…) to indicate where data was cut.
Sources: packages/scoutline/tests/output-budget-t4.test.js:358-380, packages/scoutline/tests/output-budget-t4.test.js:751-760
Artifact Storage and Recovery
Research results are saved as artifacts to prevent data loss and support reproducible lookups. The writeArtifact function saves the full, untrimmed result even if a --max-chars budget was applied to the output.
Storage Constraints
- RequestId: Generated as
<UTC compact>-<4 lowercase hex>(e.g.,20260829T142233Z-7f3a). - Atomicity: Writes are atomic to prevent corruption during concurrent research runs.
- Permissions: Artifact directories are created with
0700permissions and files with0600.
Sources: packages/scoutline/tests/artifacts-store.test.js:14-25, packages/scoutline/tests/artifacts-store.test.js:115-125
Provider Conformance
Different providers handle research controls with varying levels of support. Scoutline enforces a strict conformance policy to prevent "accept-and-drop" behavior.
- Consumed: The provider natively supports the control (e.g., Tavily supports
modelandinclude_domains). - Rejected: The provider does not support the control, and the CLI throws an
UnsupportedOptionError. - Documented Strip: The control is accepted for workflow compatibility but stripped before reaching the wire, accompanied by a stderr warning.
Sources: packages/scoutline/tests/controls-conformance.test.js:23-45, packages/scoutline/tests/controls-conformance.test.js:1100-1115
Summary
Asynchronous research provides a robust framework for long-running investigations within Scoutline. It coordinates complex task lifecycles across multiple providers, ensures output fits within character budgets, and maintains a secure, atomic artifact store for all research results. Significant emphasis is placed on ensuring that user steering controls are either fully wire-consumed or explicitly rejected.
Sources: packages/scoutline/tests/controls-conformance.test.js:12-21, scoutline-augmentation-spec.md:6-10