[codex] Fix ClickHouse activity dedupe build

Merged
AS
5mo ago
+541-321

Replaces the failing recursive ClickHouse activity dedupe graph with domain read models for source records, duplicate matches, duplicate groups, and deduped activities.
This fixes the production activities empty state caused by analytics.deduped_activities staying empty after ClickHouse hit MEMORY_LIMIT_EXCEEDED in the old recursive model.
Production has already been deployed to sha-142216b; the first analytics-worker dbt build completed successfully with 15/15 models in 28.24s and ClickHouse now reports 73 recent deduped activities.
Validated with focused read-model tests, analytics policy lint, root/server/web typechecks, production health check, and production ClickHouse row-count checks.

Summary by cubic

Rebuilt ClickHouse activity dedupe as incremental read models so builds are memory-safe and analytics.deduped_activities is repopulated. Adds deterministic matching and fixes a nullable sort key that blocked clean first builds and CI.

  • Bug Fixes

    • Replaced recursive dedupe with activity_source_records → activity_duplicate_matches → activity_duplicate_groups → deduped_activities (all ReplacingMergeTree(refresh_version)), eliminating ClickHouse MEMORY_LIMIT_EXCEEDED; dbt build succeeds and recent activities now show in production.
    • Uses a consistent null-ended 12-hour window for duplicate matching and merged bounds.
    • Ensured non-null sort key in deduped_activities by emitting assumeNotNull(user_id), fixing the ClickHouse “allow_nullable_key” error in clean e2e builds.
    • Updated tests to target the new models and assert the non-null user_id sort key; added the new models to DBT_SAFE_MODELS in entrypoint.sh.

  • Refactors

    • Deleted analytics/macros/bounded_activity_graph.sql and rewired deduped_activities to use the new read models.
    • Groups duplicates via a two-hop link walk (no recursive CTEs) with deterministic settings (max_threads: 1, join_use_nulls: 1) and deterministic device-priority tie-breakers (pattern length DESC, priority ASC, pattern ASC).
    • Expanded the production incident doc with June 1–2 root-cause, CI sort-key update, and mitigation details.

Summary by CodeRabbit

Release Notes

  • Bug Fixes

    • Resolved issue where activity deduplication could fail due to memory constraints, causing empty activity views.

  • Refactor

    • Restructured activity deduplication pipeline to improve reliability and performance through clearer separation of processing stages.