feat(knowledge): crawl Slack per member on Sim Search through each person's own Slack user token
Merged
WA
2mo ago+691+691+012345678901234567890123456789-132-132-012345678901234567890123456789
Summary
- Slack joins the per-member Sources on Sim Search. Credential Group enrollment already issues each person their own Slack user token through the workspace's custom Slack app, and
conversations.listunder that token returns exactly the public channels and the private channels they belong to, so the existing members engine crawls Slack the same way it crawls Gmail and Google Chat: each member's listing is their access, and the reader tokens:slack:<team>:<user>is derived from the same credential row on both sides. No new access mode, no bot crawl, no membership walk, no email matching. - Slack's 2025 rate-limit change to
conversations.historyexempts internal customer-built apps, which is what a Credential Group's custom bot is, so a per-member crawl runs at full Tier 3 speed. - The Slack connector now lists channels as deferred stubs: the transcript is fetched in
getDocument, once per channel per run however many members list it, and hydrates five at a time under the shared 12 MiB text ceiling. With no channel configured it lists every readable channel through the cursor, which is what a members-mode crawl does (the channel selection is the listing cap). A channel listing carries no signal of new messages, so the stub hash is run-scoped and the real hashgetDocumentcomputes decides whether anything is re-indexed: an unchanged channel costs one history page and no embedding. - Each channel document opens with a header naming the channel, its topic and its purpose, so an empty channel is a live document rather than a deletion, and the message count reports only the messages the transcript holds. The hash prefix moves to
slack-v3so channels indexed before the header and ceiling re-index once. - The shared text accumulator now takes which end survives when a document reaches the ceiling. A mail thread keeps what came first, because its root message is the context; a chat transcript keeps what came last, so the newest messages are the ones people search for. Google Chat moves to the same rule, and both keep their pinned header with a note where text was left out.
- Members-mode scope validation goes through the Credential Group provider adapter instead of the managed OAuth policy registry, which had no Slack entry. The standard OAuth adapter's scope comparison now reads the provider's policy alone, so validating a saved option no longer needs the OAuth client configured.
- First connect from Sim Search cannot create a Slack Credential Group, since the option authorizes through the workspace's own Slack app that only an admin can configure in Settings: it adopts the one ready Slack option, and otherwise says what to set up. Options that are not configured yet are never adopted, for any provider.
Type of Change
- New feature
Testing
- New Slack connector suite: deferred stubs without history reads, all-channel listing with cursor paging and the public and private channel types, one stub per channel across members of a run, a new stub each run, the header and
slack-v3hash on hydration, an empty channel kept live, null only for a channel Slack no longer knows. - Text accumulator: keeping the last records drops the oldest to fit, keeps the pinned header, refuses only a record too large on its own, and joins plainly when everything fits.
- Members-mode binding validation: a Slack option passes through the Slack scope policy, a channel selection is refused as a cap, a missing history scope is refused.
- Provisioning: adopts the one ready Slack option, points at Settings when none is ready, leaves two for the admin, and still creates a group for a standard OAuth provider.
- The cross-connector permission-scoped listing test now includes Slack and checks every connector's read scopes against its Credential Group adapter.
- Slack's
conversations.historyreference confirms user tokens read public channels they are not in, which the all-channel listing relies on. - Type-check, lint,
check:audits(45 audits), and the knowledge, credential-group, connector, and search suites (2,832 tests) pass.
Checklist
- Code follows project style guidelines
- Self-reviewed my changes
- Tests added/updated and passing
- No new warnings introduced
- I confirm that I have read and agree to the terms outlined in the Contributor License Agreement (CLA)