Bugbot Rules: Testing Constraints
Relevant source files
The following files were used as context for generating this wiki page:
Introduction
Testing constraints within the amir-bug-patterns project address a specific class of errors where the code is broken, but the test suite remains green, leading to a false sense of security in the Continuous Integration (CI) pipeline. Unlike standard bug patterns that document flaws in application logic, these constraints focus on the test itself as the source of the bug. The project emphasizes that when CI "lies," the issue cannot be caught by more tests of the same flawed type.
The scope of these constraints includes ensuring tests mimic real consumer behavior, verifying that new tests actually fail on old code, and preventing infrastructure-induced false positives like shared state between test runs. These rules are universally applicable across all technology stacks within the project, acting as a critical baseline for maintaining software quality.
Sources: TESTING-PATTERNS.md:3-12, INTEGRATION.md:148-152
Core Testing Principles
The project outlines several foundational rules that must be followed when creating or reviewing tests. These are designed to ensure that tests provide real evidence of coverage and correctness.
Consumer-Centric Validation
Tests must interact with the code through the same interface as the actual consumer. A common failure mode occurs when a test validates a specification (e.g., a direct function call or a manual POST request) while the real user interacts via generated HTML or a specific client implementation that might deviate from that spec.
Sources: TESTING-PATTERNS.md:16-30, claude-md-snippets/testing.md:3-8
The diagram shows the difference between testing a backend handler directly versus testing through the actual interface (HTML) generated for the user.
Sources: TESTING-PATTERNS.md:25-30, claude-md-snippets/testing.md:3-8
The "Failure Proof" Requirement
A new test added alongside a fix must be verified to fail on the old code. If the assertion passes even in the broken state (e.g., checking for an empty result that is "correctly" empty in both scenarios), the test does not provide valid evidence of the fix. This is particularly critical for race conditions and security flaws.
Sources: TESTING-PATTERNS.md:95-105, claude-md-snippets/testing.md:23-26
Testing Constraint Categories
The following table summarizes the key constraint categories documented in the repository:
| Constraint ID | Title | Description | Severity |
|---|---|---|---|
| T1 | Spec vs Consumer | Test fails because it uses an interface the real consumer does not. | HIGH |
| T2 | Failure Verification | Test must be proven to fail on the original broken implementation. | MEDIUM/STRICT |
| T3 | Infra Shared State | Infrastructure (mocks, fixtures) leaking state between test runs. | LOW |
| Sources: TESTING-PATTERNS.md:16, TESTING-PATTERNS.md:95, TESTING-PATTERNS.md:118 |
Environment and Enforcement Gaps
Tests conducted using standard libraries like httpx or requests do not enforce browser-side security policies. Features relying on Content Security Policy (CSP), SameSite cookies, CORS, or postMessage origins require testing in a real browser environment (e.g., Playwright) because the server-side test client will not see the "silent block" imposed by the browser.
Sources: TESTING-PATTERNS.md:36-45, claude-md-snippets/testing.md:10-14
This diagram illustrates how a standard test client might return a successful response while a real browser blocks the same action due to security policies.
Sources: TESTING-PATTERNS.md:36-45, claude-md-snippets/testing.md:10-14
Concurrency and Atomicity
Tests for atomicity, locks, or rate limits must be run in parallel (e.g., using asyncio.gather or multiple threads). Sequential calls do not prove thread safety or idempotency, as they do not simulate the race conditions found in production.
Sources: TESTING-PATTERNS.md:47-49, claude-md-snippets/testing.md:16-18
Infrastructure Best Practices
Errors in how the test framework is used can lead to confusing failures or session management bugs.
- Fixture Imports: Developers must not import fixtures from
conftest.py. Pytest registers these automatically; manual imports can cause fixtures withscope="session"to run multiple times, leading to singleton manager failures. - State Reset: Mocks and singletons with internal state must be reset during the fixture teardown phase to prevent non-deterministic failures where a test only fails if run in a specific order.
Sources: TESTING-PATTERNS.md:121-130, claude-md-snippets/testing.md:28-31
Code Snippet: Pytest Fixture Anti-Pattern
# BAD: Manual import of a fixture
from tests.conftest import mcp_lifespan
# GOOD: Rely on pytest discovery
@pytest.mark.asyncio
async def test_feature(mcp_lifespan):
# mcp_lifespan is injected automatically
...
Sources: TESTING-PATTERNS.md:123-129, claude-md-snippets/testing.md:28-31
Conclusion
Testing constraints in the amir-bug-patterns repository serve as a safeguard against "lying" CI suites. By enforcing that tests mirror real-world consumer interfaces, verifying failures against original code, and strictly managing test infrastructure state, the project ensures that green lights in CI represent genuine system health. These rules are prioritized in code reviews, especially for critical paths involving security, concurrency, and external integrations.
Sources: TESTING-PATTERNS.md:3-12, INTEGRATION.md:131-140