Add deterministic catalog export CSV parser for Context imports

SI
13d ago
+1500-0

Summary

Adds a pure, deterministic, side-effect-free parser module at lib/context/imports/ that turns the UTF-8 text of a generic catalog export CSV into versioned proposals with row/line/field pointers. This is the first unaddressed bounded-acceptance item of #2140 ("a deterministic parser for an actual catalog export … preserve row/page/field pointers and parser version; handle duplicate/versioned imports, conflicts and malformed records … preserve extracted claims as proposals distinct from reviewed assertions") and follows the epic's "manual and deterministic parsing first" direction. The module has no database, HTTP/MCP, storage, provider, environment, clock or randomness access and adds no npm dependency; it sits beside the open originals/evidence PRs and touches none of their files.

  • parseCsvRecords.ts (+ csvRecordTypes.ts, findClosingQuote.ts): a hand-written RFC 4180 tokenizer that handles quoted fields, doubled quotes, CRLF/LF, embedded newlines and a BOM. empty_row, column_count_mismatch, unterminated_quote and text_after_closing_quote records are reported with pointers instead of being dropped or repaired. A cell with text after its closing quote keeps its literal source text, quotes included. A quote that nothing later can close is reported at its opening line, however long the remaining input is. Trailing blank lines are not records. Hard caps of 10,000 data rows, 1,000 columns per record and 10,000 characters per cell throw rather than truncate.

  • parseCatalogExport.ts (+ buildCatalogExportProposal.ts): the parser emits parserVersion: catalog-export-csv-v1 and profileVersion: generic-catalog-export-v1. It adds a SHA-256 contentFingerprint of the decoded input text; this is a parse-input fingerprint and not the original's byte hash, as the README and schema JSDoc state. It also adds a headerFingerprint of the verbatim header. Every proposal is literally status: "proposed". ISRC and UPC values are shape-checked only, with these states:

    • present
    • malformed, with the raw text kept
    • unknown, for an empty cell
    • uncollected, when the export has no recognised column

    Values are never generated, padded or repaired. same_isrc and same_row_content duplicates are reported in first-row order, and no rows are dropped. Instruction-like cell text is carried as a plain string. Tab- or semicolon-delimited text is refused rather than parsed into an all-unmapped result.

  • resolveCatalogExportField.ts: a small, distributor-neutral header alias set. A bare Title, Song or Song Title stays unmapped, so release or composition titles are not proposed as track titles. Unknown columns are kept verbatim, and when two columns collide on an alias the first one wins.

  • catalogExportTypes.ts: Zod schemas, including a discriminated union so each identifier state allows only its own value/raw shape. ContextImportError.ts: typed whole-input error codes (unsupported_input, empty_input, header_missing, row_limit_exceeded, column_limit_exceeded, cell_limit_exceeded). A thrown error never comes with a partial result.

  • lib/context/imports/README.md documents three things: what the parser does not do (import, review, link, identity, rights, grain), what it detects as unsupported, and where a later slice connects it.

Refs recoupable/app#2140 (child of epic recoupable/app#2116).

Merge sequencing

docs → database → api → app. This PR touches only the api and stands alone: no docs contract change, no migration, no app change, and no new action registered in processContextOperation.ts, contextToolOperations.ts or tool-policy.json. It does not depend on or modify any OPEN PR, and is designed to sit beside api#983/#985/#986–#991 and database#90/#92/#94/#95. None of their files are touched, and lib/context/README.md is deliberately left unedited because sibling PRs modify it. A follow-up slice would read a retained original (api#987/#990 receipts) as text and call parseCatalogExport. It would then persist proposals against that exact source version, keyed on the original's byte SHA-256 and not contentFingerprint. That persistence needs a new owning-database migration, which must land before any HTTP/MCP exposure.

Verification

All results below are from the head commit c7b91add.

  • Ran pnpm exec vitest run lib/context/imports: 2 files, 43 passed / 0 failed / 0 skipped.
    • 17 tokenizer cases: pointers, CRLF, quoted commas, embedded newlines, BOM, header-only input, interior vs trailing blank lines, column-count mismatch, unterminated quote (short, over the cell cap, and on a realistic 3,000-row tail), text after a closing quote, closed long quoted cell, and the row, column and cell caps.
    • 26 catalog parser cases. Rows and fingerprints: byte-identical replay, changed-cell fingerprints, duplicate ISRC/row and cross-kind ordering, row cap. Identifiers: malformed vs unknown vs uncollected, the space/hyphen-only normalisation, and schema rejection of contradictory identifier states. Columns: alias headers, ambiguous bare Title/Song, unmapped columns. Input checks: instruction text kept as data; rejection of PDF/zip (with and without BOM), binary text, C1 controls, unpaired surrogates and tab/semicolon delimiters. Also a schema round-trip and module purity.
  • Ran pnpm exec vitest run lib/context: 69 files passed / 8 skipped, 369 tests passed / 8 skipped (377). The baseline on origin/main c72887a was 67 files / 326 tests passed, 8 skipped. No live tests were executed, and neither CONTEXT_LIVE_TEST nor CONTEXT_PAID_TEST was set.
  • Ran pnpm exec tsc --noEmit: exits 2 with the same 199 pre-existing errors in files this PR does not touch, and 0 errors in lib/context/. That includes 0 in lib/context/imports/.
  • Ran pnpm exec eslint lib/context/imports: clean. pnpm exec prettier --check "lib/context/imports/**/*.ts": clean.
  • No new dependency: package.json and pnpm-lock.yaml are unchanged.
  • Red → green (initial commit): both test files were written before any module existed and failed with Cannot find module.
  • Red → green (review fixes): 16 new cases were added, and 2 existing schema-summary expectations and 1 header expectation were updated. Before the fix, 17 of 43 failed: text after a closing quote, unterminated quote over the cell cap, long quoted cell line, trailing blank lines, column cap, uncollected state/summary, \s normalisation, ambiguous Title, delimiter refusal, duplicate ordering, identifier union, and BOM-PDF/C1/surrogate rejection. All 43 pass after the fix.
  • Not verified: hosted deployment, any live provider, production or preview database, or any real customer or distributor export. The fixtures are synthetic, with ZZTST… ISRCs and zero-padded UPCs, and the alias profile has not been calibrated against a real export. A hosted roundtrip through the disabled api#991 intake was not run.
  • Commit signatures: the original commit 552e77b is unsigned; the 1Password SSH signer failed at the time, and this is disclosed. The follow-up commit c7b91add is SSH-signed. main currently has no ruleset requiring signatures. If one is added, re-sign the first commit before merging.

Review follow-up

Changes made after the independent review and the cubic review:

  • Fixed:
    • Text after a closing quote is no longer silently joined onto the cell.
    • An unterminated quote in a large file is reported at its own line instead of as cell_limit_exceeded.
    • Trailing blank lines are not reported as empty_row.
    • A missing identifier column is now uncollected, kept separate from an empty unknown cell.
    • \s normalisation narrowed to spaces and hyphens.
    • Bare Title/Song no longer guesses the row's grain.
    • Contradictory identifier states are rejected by the schema.
    • Unbounded column count is now capped.
    • BOM-prefixed PDF/zip, unpaired surrogates and C1 controls are rejected.
    • Duplicate groups are ordered by first row across kinds.
    • The README's unsupported-format and fingerprint claims are corrected.
    • Invisible BOM/U+FFFD literals replaced with escapes.
    • Error class file renamed to PascalCase (ContextImportError.ts), and the column module renamed to match its function (resolveCatalogExportField.ts).
  • Partly addressed: the "files under 100 lines" note. parseCatalogExport.ts is now 93 lines after extracting buildCatalogExportProposal.ts, and the tokenizer's types moved to csvRecordTypes.ts. parseCsvRecords.ts (179 lines) is still one single-pass state machine, kept together for cohesion. catalogExportTypes.ts (131 lines) is a schema-only file. No 100-line rule appears in this repo's AGENTS.md/CLAUDE.md, and several existing lib/context modules exceed it (for example processContextOperation.ts at 511 lines and planStoredContextModules.ts at 240).

Out of scope / known gaps

  • Persisting proposals needs a new owning-database migration after database#94/#95. The 202610102140xx migration slot is left free for it.
  • HTTP/MCP exposure. No action is added to the shared Context operation.
  • Attaching proposals to retained originals or to person/company/recording/composition/release subjects (api#983/#987). Deciding a row's grain (recording vs release vs composition) is left to that later reviewed step.
  • Calibrating the header profile against a real (de-identified) distributor export. This needs a sample supplied by a human.
  • Reviewing a proposal into an assertion, staling only the affected reviews when a new version arrives, and resolving conflicts across versions. This PR only supplies the fingerprints and pointers those steps need.
  • Formats that are not supported: PDF, images/OCR, DDEX, royalty and sales statements, XLSX, non-comma delimiters, non-UTF-8 encodings, and paid model extraction. Only the signatures and characters listed in the README are detected, so callers must bound and type-check originals before decoding them.
  • Identifier check-digit validation. Checks are shape-only by design.
  • Activation of the disabled original intake (api#991).

🤖 Generated with Claude Code