Keep the header when a body cell repeats its column label

Merged
CH
2d ago
+196-8

Slack thread: https://firecrawl.slack.com/archives/C08LS3XTELB/p1792189170824319

Problem

For CSV/xlsx/ods (no declared header row), resolve_header_rows counted a "data" vote whenever a text column's first-row value recurred in the body (case-insensitive, trimmed). That vote tied with real type evidence, e.g. a numeric column under a text label, so detection returned 0: the output got a blank header row and the real labels were pushed into the body.

Repro: city,region,pop / Paris,north,100 / Lyon,region,200 / Nice,east,300 rendered as | | | | followed by | city | region | pop | in the body.

Fix

Repeated first-row values now go into a separate repeat_votes tally:

  1. Any type evidence decides on its own (header_votes > data_votes).
  2. If there is no type evidence, any repeat still means no header, so real headerless tables (x,aka / y,aka / z,aka) are unchanged.
  3. Otherwise the existing row-shape fallback applies.

Behaviour change: a table with type evidence for a header plus two or more repeated values now also keeps its header. Before, the repeats outvoted it.

Tests

New tests/header_collision.rs builds each case in memory as CSV, xlsx and ods. For every case and format it asserts that the first output line is the header (| city | region | pop |, or | city | Status | pop | for the Status case) and that the table has data rows + 2 lines. Cases: collision in col A, collision in col B, upper case, mixed case, padded, collision on the first data row, Status containing status, and no collision. There is also an exact-output test for the no-collision case, which passes on the old code too.

cargo test (all tests plus the snapshot corpus, no snapshot changes), cargo clippy --all-targets -- -D warnings and cargo fmt --check all pass.

Generated by Claude Code

Summary by cubic

Fixes header detection so a body cell that repeats its column label no longer overrides real type evidence. Previously a repeat counted as a vote against a header, so tables like city,region,pop with a body row containing Lyon,region,200 rendered with a blank header row and the labels pushed into the body; now type evidence decides on its own, and repeats only settle tables that have no type evidence.

Behavior change: a table with type evidence for a header plus two or more repeated labels now keeps its header, where the repeats used to outvote it. Adds tests/header_collision.rs covering collisions across CSV, xlsx, and ods.