feat: implement issue #2504 — Reviewer scorecard: classify each bot message so refusals, skips and status notices stop counting as reviews
User description
Problem
Reviewer scorecard: classify each bot message so refusals, skips and status notices stop counting as reviews
From the issue: The weekly reviewer scorecard (scripts/reviewer_report.sh, posted to #1220) can't tell a refusal from a review. Each bot message is checked against one shared pattern, ADVISORY_RATE_LIMIT_RE (from scripts/lib/advisory-review-gate.sh). A message that matches is a refusal; every other message counts as a real review. For a bot with no formal PR review, the issue-comment fallback in _NORMALIZE_JQ also turns its status, skip and quota comments into "reviews".
Risk
Low — changes automation shell logic under scripts/, covered by shellcheck (--severity=warning) and the bats suite.
Test plan
Tests added/updated: tests/fixtures/reviewer_message_classes.json,tests/fixtures/reviewer_pr_node.json tests/reviewer_report.bats. Verification: bash scripts/dev-lead-lint.sh (shellcheck --severity=warning) ran pre-commit; the bats suite runs in CI.
Rollback
Revert this PR. No non-revertible side effects (no tags, migrations, or external state).
Monitoring
This PR's Lint (shellcheck) and bats checks show pass/fail; watch subsequent dev-lead / pr-review runs for behavioral regressions.
Closes #2504
CodeAnt-AI Description
Separate actual bot reviews from notices in the reviewer scorecard
What Changed
- Classifies bot messages as findings, clean reviews, refusals, skips, status notices, or unclassified. Only findings and clean reviews count as reviewed, so notices and refusals no longer inflate review and coverage counts.
- Separates the scorecard into findings, clean reviews, refusals, skips, and no response, and adds findings raised and resolved-thread percentages. Event counts now appear in a Details section.
- Shows up to three short examples of unclassified messages per bot so new message formats are visible.
Impact
✅ Fewer status notices counted as reviews
✅ Separate counts for refusals and skipped reviews
✅ Visible examples of unrecognized bot messages
💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
@codeant-ai ask: Your question here
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
@codeant-ai ask: Can you suggest a safer alternative to storing this secret?
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
@codeant-ai: Your feedback here
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
@codeant-ai: Do not flag unused imports.
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
@codeant-ai: review
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.