Session JSONL Parser

Relevant source files

The following files were used as context for generating this wiki page:

• src/parser.ts
• src/types.ts
• src/parser.test.ts
• src/cli.ts
• src/format.ts
• AGENTS.md

The Session JSONL Parser is a core component of the cxs system responsible for ingestion and transformation of local Codex session logs into structured data. It processes .jsonl files (JSON Lines) to extract session metadata, message transcripts, and specialized summaries used for full-text search (FTS) and session ranking. Sources: src/parser.ts, AGENTS.md:27-28

The parser acts as the gateway between raw log files in ~/.codex/sessions and the SQLite indexing engine. It specifically targets human-AI interaction by filtering internal system markers and prioritizing user/agent messages while aggregating "compacted" handoffs and reasoning summaries into session-level metadata. Sources: src/parser.ts:5-13, README.md:120-125

Core Architecture and Logic

The parser uses a stream-based approach to handle large session files efficiently, reading line-by-line and maintaining an internal state during the parse operation. Sources: src/parser.ts:25-45

Parsing Flow

The parsing logic follows a strict sequence:

1. Stream Initialization: Opens a read stream for the target .jsonl file.
2. Line Processing: Iteratively parses each JSON line into a record.
3. Type Dispatching: Routes records based on their type field (e.g., session_meta, event_msg, compacted).
4. Filtering: Removes internal agent markers and "synthetic" messages.
5. Aggregation: Computes session bounds (start/end times) and generates a summaryText.

Data Structures

The parser interacts with several key TypeScript interfaces to ensure type safety and structured output.

Input Record Types

The parser identifies and processes specific JSON structures within the log files:

Record Type	Payload Fields	Purpose
session_meta	id, cwd	Extracts Session UUID and working directory.
turn_context	model, cwd	Captures AI model version and context path.
event_msg	type, message	Extracts user/agent dialogue messages.
compacted	message	Captures durable state or "handoff" summaries.
response_item	type: "reasoning", summary	Extracts AI reasoning steps/summaries.

Sources: src/parser.ts:86-125, src/parser.test.ts:22-35

Output: ParsedSession

A successful parse results in a ParsedSession object, which contains all data needed for indexing.

export interface ParsedSession {
  sessionUuid: string;
  filePath: string;
  title: string;
  summaryText: string; // Combined first/last messages
  compactText: string; // Unique 'compacted' messages
  reasoningSummaryText: string;
  cwd: string;
  model: string;
  startedAt: string;
  endedAt: string;
  messages: ParsedMessage[];
}

Sources: src/types.ts:13-25

Specialized Parsing Rules

Message Filtering and Internal Markers

To maintain a clean search index, the parser ignores messages that match specific internal Codex markers. These markers usually indicate approval requests or transcript headers that do not represent actual dialogue. Sources: src/parser.ts:5-13

Key markers include:

• >>> TRANSCRIPT START
• >>> APPROVAL REQUEST START
• The following is the Codex agent history...

If a session contains only filtered messages, it is categorized as kind: "filtered" and is not indexed. Sources: src/parser.ts:63, src/parser.test.ts:114

Summary Generation Logic

The parser generates a summaryText for every session by extracting and normalizing the first and latest messages from both the user and the assistant. Sources: src/parser.ts:153-176

Session Title Extraction

The "Title" of a session is derived from the first user message, truncated to the first 120 characters. If no user message is found, the title defaults to (no title). Sources: src/parser.ts:66, src/parser.ts:147-151

Performance Optimizations

The parser implements specific O(N) optimizations to handle large log files without unnecessary memory overhead:

• Timestamp Tracking: Identifies session start (minTimestamp) and end (maxTimestamp) in a single pass over the messages, avoiding multiple array allocations or sorting operations. Sources: src/parser.ts:69-80
• Normalization: Uses normalizeUniqueText with a Set to deduplicate compacted messages and reasoning summaries during parsing. Sources: src/parser.ts:186-196
• Reverse Search: When building summaries, the parser loops through messages in reverse to find the "latest" messages efficiently. Sources: src/parser.ts:161-167

Error Handling and Result Kinds

The parseCodexSession function returns a tagged union ParseSessionResult to signal different outcomes to the indexer. Sources: src/types.ts:27-31

Kind	Trigger Condition	System Action
parsed	Valid Session UUID and at least one non-filtered message found.	Proceeds to SQLite indexing.
filtered	Messages were found, but all were internal markers.	Ignored by indexer.
skipped	No UUID found or no event messages present.	Ignored by indexer.

Sources: src/parser.ts:63-64, src/types.ts:27-31, src/parser.test.ts:114

Conclusion

The Session JSONL Parser is engineered for high-fidelity extraction of local Codex history. By distinguishing between dialogue (event_msg) and metadata (compacted, reasoning), it enables the cxs system to provide weighted search results where titles and handoff summaries carry more relevance than individual message snippets. Its stream-based processing and internal marker filtering ensure that the resulting index remains both compact and focused on human-relevant content. Sources: AGENTS.md:27-32, src/parser.ts:86-125