Testing Fallacies (T1-T3)
Relevant source files
The following files were used as context for generating this wiki page:
Testing Fallacies (T1-T3) refers to a specific axis of bug patterns where the code implementation might be technically functional, but the tests fail to validate reality. This creates a state where the Continuous Integration (CI) system "lies," showing green results while the feature is broken in production. Unlike other patterns in the project (such as Core Patterns or Critical Patterns), which focus on code defects, these fallacies address defects in the verification layer itself.
Sources: TESTING-PATTERNS.md:3-11, README.md:162-166
T1: Spec Validation vs. Consumer Reality
The most significant fallacy occurs when a test validates against a technical specification or internal interface rather than the actual consumer interface. This results in "false security" where the logic passes unit or integration tests, but fails when accessed by a real user or external client.
Sources: TESTING-PATTERNS.md:16-21, claude-md-snippets/testing.md:3-8
Disconnect Archetypes
The following diagram illustrates the gap between a "Green CI" and a "Broken Production" state caused by interface mismatch.
Diagram showing the divergence between manual test calls and actual user interaction flows. Sources: TESTING-PATTERNS.md:27-38, BY-STACK/browser-policy.md:30-38
| Fallacy Variation | Description | Project Impact |
|---|---|---|
| Manual vs. Handler | Test sends manual POSTs; user uses a generated HTML form. | Broken forms pass tests. |
| Spec vs. Client Path | Test uses /mcp/ (trailing slash); real client uses /mcp. | Redirects (307) break auth. |
| Environment Enforcement | Test tools (httpx/requests) ignore browser policies. | CSP/CORS failures missed. |
| Serial vs. Parallel | Test calls methods sequentially to check atomicity. | Race conditions missed. |
Sources: TESTING-PATTERNS.md:27-46, claude-md-snippets/testing.md:10-18
Real-World Examples
- Markdown-Docs #20: The registration endpoint was tested based on the specification. However, the real client (claude.ai) sent requests to
/registerat the root, leading to a 404 in production despite passing tests. - Markdown-Docs #21: A test sent a
POSTdirectly to an endpoint. In a browser, the action was blocked by aform-action 'none'CSP policy. The test was "Green," but the button was "Dead."
Sources: TESTING-PATTERNS.md:52-59
T2: Absence of Failure Evidence
A test added alongside a fix is not evidence of coverage if it hasn't been proven to fail without the fix. This is particularly critical for race conditions, security policies, and idempotency locks where the probability of the test failing to reproduce the failure condition is high.
Sources: TESTING-PATTERNS.md:98-103, claude-md-snippets/testing.md:24-27
Sequence for validating test effectiveness. Sources: TESTING-PATTERNS.md:118-125
Mandatory Verification Rule: For race/idempotency tests, developers must document that the test resulted in multiple successes (e.g., [200, 200]) before the fix and a single success (e.g., [200, 400]) after.
Sources: TESTING-PATTERNS.md:106-110, INTEGRATION.md:179-183
T3: Infrastructure-Induced Confusion
The testing framework or infrastructure itself can become a source of non-deterministic failures or misleading errors, often burning debugging hours on phantom bugs.
Sources: TESTING-PATTERNS.md:133-137
Common Infrastructure Pitfalls
- Fixtures Import: In
pytest, importing fixtures fromconftest.pymanually (e.g.,from tests.conftest import fixture_x) is strictly forbidden. This can causescope="session"fixtures to register and run multiple times, leading to errors like "session manager can only be called once." - Shared State: Mocks or singletons that are not reset between tests. This leads to tests that pass individually but fail when run in a specific order or as part of a full suite.
Sources: TESTING-PATTERNS.md:141-155, claude-md-snippets/testing.md:29-32
Detection Signature for Bugbots
Identify:
1. 'from ... conftest import' of a function decorated with @pytest.fixture.
2. Mocks/singletons lacking a 'reset' in fixture teardown.
3. Feature tests using manual requests instead of deriving inputs from generated HTML/components.
Sources: TESTING-PATTERNS.md:65-79, TESTING-PATTERNS.md:159-163, INTEGRATION.md:185-187
Summary
Testing Fallacies (T1-T3) serve as a reminder that a green test suite is not synonymous with a working system. By enforcing "Strict" mode for external consumer flows (OAuth, webhooks, public APIs) and requiring browser-based verification (e.g., Playwright) for client-side policies, the project mitigates the risk of CI lying about deployment readiness. The most critical action is ensuring tests fail on buggy code before they are trusted to validate fixes.
Sources: TESTING-PATTERNS.md:86-93, MIGRATION-NOTES.md:102-106