fix(knowledge): make indexing and connector recovery durable

Merged
IC
1mo ago
+29238-535

Summary

  • Save workspace imports as immutable KB-owned files before queueing, using the shared upload attachment and orphan-cleanup guard.
  • Recover abandoned stored connector documents independently of source sync. Atomically install a new processing generation and outbox delivery with bounded attempts, runtime, and admission backoff.
  • Treat shared provider capacity as a durable deferral. Preserve content and member progress across GitHub waits, pin immutable tree/blob reads, and reconcile removals only after a complete listing.
  • Preserve safe, structured source failures and retry only positively identified transient database reads. Set HTTP body framing in the shared pinned transport.
  • Keep regular KB and organization Search ownership, permissions, complete-index publication, and billing behavior intact. Apply the additive migration before deploying the workers.

Type of Change

  • Bug fix

Testing

  • 4,080 unit tests passed across affected knowledge, connector, processing, storage, deadline, and provider-capacity paths.
  • 866 shared HTTP security tests passed, including real-socket body framing and redirect regressions.
  • 328 integration tests and 13 PostgreSQL access/search tests passed in disposable PostgreSQL/Redis, including synthetic content against live OpenAI/Mistral APIs. The 42 skipped tests require separate opt-in provider or scale fixtures.
  • Type check, lint, all 46 CI audits, API validation, docs manifest, block registry, and migration safety checks pass.
  • Recovery coverage includes concurrent admission, duplicate/stale delivery, atomic rollback, blocked owners, organization Search authorization, immutable imports after source deletion, and concurrent-index migration replay.

Checklist

  • Code follows project style guidelines
  • Self-reviewed my changes
  • Tests added/updated and passing
  • No new warnings introduced
  • I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)