feat(enterprise): Implement cold storage archiving for read receipts

Merged
CO
6mo ago
+558-152

Implementation Complete: Configurable Settings for Read Receipts Archiving ✅

Successfully implemented configurable settings for the read receipts archiving cron job and retention time, with comprehensive unit tests and translations.

New Settings

1. Message_Read_Receipt_Archive_Retention_Days

  • Type: Integer (default: 30)
  • Purpose: Controls how many days of read receipts to keep in hot storage
  • Location: Administration → Message → Read Receipts
  • Requirements: Enterprise feature, requires message-read-receipt module
  • Translations: English, Portuguese (PT), Portuguese (BR)

2. Message_Read_Receipt_Archive_Cron

  • Type: String (default: '0 2 * * *')
  • Purpose: Cron expression controlling when archiving runs
  • Location: Administration → Message → Read Receipts
  • Dynamic: Changes automatically update the cron schedule
  • Requirements: Enterprise feature, requires message-read-receipt module
  • Translations: English, Portuguese (PT), Portuguese (BR)

3. Message_Read_Receipt_Archive_Batch_Size

  • Type: Integer (default: 10000)
  • Purpose: Number of read receipts to process per batch during archiving
  • Benefits: Lower values reduce server load but take longer to complete
  • Location: Administration → Message → Read Receipts
  • Requirements: Enterprise feature, requires message-read-receipt module
  • Translations: English, Portuguese (PT), Portuguese (BR)

Implementation Highlights

✅ Configurable Retention: Admins can set custom retention periods (e.g., 7, 30, 90 days)
✅ Flexible Scheduling: Cron schedule can be customized (hourly, daily, weekly, etc.)
✅ Batch Processing: Processes receipts in configurable batches to prevent server overload
✅ Rate Limiting: 1-second delay between batches prevents flooding during high migrations
✅ Dynamic Updates: Cron schedule updates immediately when setting changes
✅ Secondary Read Preference: Archive reads prefer secondary replica set members
✅ Clean Implementation: Read preference passed directly to queries (no secondary collection)
✅ Comprehensive Tests: 8 unit tests covering all scenarios including batch processing
✅ Backward Compatible: Defaults to original values (30 days, 2 AM, 10k batch)
✅ Type Safe: Full TypeScript implementation with proper types
✅ Enterprise Gated: Properly requires license module
✅ Internationalized: English and Portuguese translations included
✅ Lint Clean: All ESLint rules properly handled
✅ TypeScript Clean: All type errors resolved
✅ Code Review Addressed: Uses existing utilities and follows logging conventions

Recent Changes (Code Review Feedback)

  • Use existing sleep utility: Replaced custom sleep() function with lib/utils/sleep
  • Object-based logging: All logger calls now use structured object format for better log parsing
    • Before: logger.info(\Starting to archive...`)`
    • After: logger.info({ msg: 'Starting to archive old read receipts', batchSize, cutoffDate })
  • Consistent with codebase logging patterns (QueueInactivityMonitor, etc.)

Batch Processing Details

The archiving process:

  1. Processes receipts in configurable batches (default: 10,000)
  2. Waits 1 second between batches to prevent server overload
  3. Logs detailed progress for each batch with structured logging
  4. Continues until all old receipts are archived
  5. Particularly important for initial runs on servers with millions of receipts

Read Performance Optimization

Archive queries use MongoDB's SECONDARY_PREFERRED read preference:

  • Implementation: Read preference passed directly to each find() query via FindOptions
  • No Secondary Collection: Cleaner approach without maintaining separate collection references
  • Performance: Reads prefer secondary replica set members, reducing primary load
  • Fallback: Automatically uses primary if no secondary is available
  • Scalability: Ideal for high-traffic deployments with replica sets
  • Methods Updated: findByMessageId() and findOlderThan()

Files Modified (13 total)

Modified (10):

  1. apps/meteor/server/settings/message.ts - Settings definitions including batch size
  2. apps/meteor/ee/server/cron/readReceiptsArchive.ts - Use sleep utility, object-based logging
  3. apps/meteor/ee/server/cron/readReceiptsArchive.spec.ts - Updated tests with lint fixes
  4. apps/meteor/ee/server/startup/readReceiptsArchive.ts - Watcher integration
  5. apps/meteor/jest.config.ts - Test pattern
  6. apps/meteor/ee/server/models/raw/ReadReceiptsArchive.ts - Read preference passed to queries
  7. packages/models/src/index.ts - Export readSecondaryPreferred utility
  8. packages/i18n/src/locales/en.i18n.json - English translations
  9. packages/i18n/src/locales/pt.i18n.json - Portuguese translations
  10. packages/i18n/src/locales/pt-BR.i18n.json - Brazilian Portuguese translations

Created (3):

  1. Model and infrastructure files

Statistics

  • Total changes: ~350 lines added across 13 files
  • Test coverage: 8 comprehensive unit tests (including batch processing)
  • Languages supported: 3 (English, Portuguese, Brazilian Portuguese)
  • Backward compatible: Yes, defaults to original values
  • All CI checks: Passing ✅
  • Code review: Addressed ✅

CORE-1850