Testing Strategy
Relevant source files
The following files were used as context for generating this wiki page:
- apps/scratch/src/pages/tests/Edit.spec.tsx
- apps/scratch/src/pages/tests/Show.spec.tsx
- apps/scratch/src/pages/tests/Solve.spec.tsx
- e2e/tests/task/task-management.spec.ts
- e2e/tests/users/user-management.spec.ts
- apps/scratch/src/pages/tests/ShowOnlyCode.spec.tsx
- e2e/tests/sessions/session-analysis.spec.ts
- e2e/tests/sessions/session-management.spec.ts
- e2e/helpers.ts
- e2e/tests/task/solve-task.spec.ts
- frontend/src/components/task/TaskForm.tsx
- frontend/src/pages/task/[taskId]/edit.tsx
- e2e/tests/task/task-management.ts
- apps/scratch/src/pages/tests/page-objects/test-task.ts
- e2e/playwright.config.ts
- e2e/tests/classes/class-management.spec.ts
- frontend/src/pages/class/[classId]/session/[sessionId]/task/[taskId]/solve.tsx
- frontend/package.json
- apps/scratch/src/pages/tests/helpers.ts
- e2e/tests/sessions/session-management.ts
The project employs a comprehensive testing strategy centered around Playwright for end-to-end (E2E) and component testing, supplemented by Jest for unit tests. This strategy emphasizes test isolation, particularly for E2E tests, which run against a full, dedicated application stack including a unique, cloned database for each test worker. This ensures that tests are reliable, independent, and do not interfere with each other.
The testing approach is divided into distinct layers: full-system E2E tests that validate user flows across the entire application, and focused component tests that verify the functionality of individual UI components, especially the complex embedded Scratch editor. This multi-layered approach ensures both high-level system integrity and low-level component correctness.
Testing Frameworks and Tools
The project utilizes several frameworks and tools to implement its testing strategy.
| Tool | Usage | Source |
|---|---|---|
| Playwright | The primary framework for both E2E and Component Testing. | frontend/package.json:52 |
| Jest | Used for unit testing. | frontend/package.json:49 |
playwright-test-coverage | A library that extends Playwright to collect code coverage. | frontend/package.json:71 |
nyc | Used for generating code coverage reports. | frontend/package.json:101 |
| TypeScript | All tests are written in TypeScript. | e2e/helpers.ts, apps/scratch/src/pages/__tests__/Edit.spec.tsx |
Sources: frontend/package.json, e2e/helpers.ts, apps/scratch/src/pages/tests/Edit.spec.tsx
End-to-End (E2E) Testing
E2E tests validate complete user workflows by interacting with the application as a real user would. These tests run against a full stack, including the frontend, backend, and a dedicated database.
E2E Test Environment Setup
A sophisticated setup process ensures that each E2E test run is completely isolated. A custom Playwright worker fixture manages the entire lifecycle of the test environment.
The following diagram illustrates the setup process for each test worker.
This diagram shows the automated setup and teardown for an isolated E2E test environment for each Playwright worker.
Sources: e2e/helpers.ts:81-224, e2e/playwright.config.ts:30-38
Key aspects of the environment include:
- Database Isolation: Each worker creates a fresh database by cloning a template, ensuring no data conflicts between parallel test runs. The template database is seeded before the main test suite runs.
- Independent Services: Each worker runs its own instance of the frontend and backend servers on dynamically allocated free ports.
- Configuration: The
playwright.config.tsfile defines thewebServercommand that orchestrates the initial seeding and setup, and specifies project dependencies to ensure setup tasks (like authentication) run before the actual tests.
Sources: e2e/helpers.ts:90-104, e2e/playwright.config.ts:80-112
E2E Test Structure and Patterns
E2E tests are organized by feature (e.g., classes, users, tasks) and follow common patterns to improve maintainability.
- Page Object Model (POM): UI interactions are abstracted into Page Models (e.g.,
UserListPageModel,TaskFormPageModel). This separates test logic from UI selectors, making tests more readable and easier to update when the UI changes. - Helper Functions: Common actions and setup routines, such as creating a new class or task, are encapsulated in helper functions (
createClass,createTask). This promotes code reuse and simplifies test cases. - Authentication: A helper
useAdminUseris used to set the authentication state in the browser context, allowing tests to start from a logged-in state.
The sequence diagram below shows a typical E2E test for creating a new task.
This diagram illustrates the interaction between the test script, Page Object Model, and the application during an E2E test.
Sources: e2e/tests/task/task-management.spec.ts:25-31, e2e/tests/task/task-management.ts:15-44
Component Testing
Component tests focus on individual parts of the application in isolation. In this project, Playwright's component testing feature is heavily used for the embedded Scratch application (apps/scratch).
Scratch Component Test Setup
Tests for the Scratch editor pages (/edit, /solve, /show) are designed to verify functionality within an iframe. A key part of this setup is mocking the communication channel between the embedded app and its parent window.
- Mocking
window.parent.postMessage: AbeforeEachhook intercepts calls towindow.parent.postMessageand stores the messages in a globalwindow.postedMessagesarray. This allows tests to assert that the component sends the correct messages to its host. - Simulating Host Events: Tests simulate events from the parent window by dispatching a custom
MockMessageEvent. This triggers handlers within the component, allowing verification of its response to host commands likegetTaskorloadTask.
Sources: apps/scratch/src/pages/tests/Edit.spec.tsx:20-33, 40-47, apps/scratch/src/pages/tests/helpers.ts:24-43
The following diagram shows how a component test simulates loading a task.
This diagram shows the flow of a component test for the embedded Scratch app, highlighting the mocking of parent window communication.
Sources: apps/scratch/src/pages/tests/Edit.spec.tsx:20-33, apps/scratch/src/pages/tests/helpers.ts:24-43
Scratch Component Test Examples
The component tests cover a wide range of user interactions within the Scratch editor:
- Block Configuration: Testing the ability to update block limits (e.g., allowing a block to be used 5 times or an infinite number of times).
- Block Manipulation: Verifying that users can (or cannot) drag, drop, and delete blocks based on whether they are frozen, editable, or appendable.
- Assertion Logic: Testing the custom assertion blocks, including running tests, checking pass/fail counts, and reporting results via
postMessage. - State Management: Ensuring the Scratch VM state is correctly reset between runs when in assertion mode, but persists otherwise.
Sources: apps/scratch/src/pages/tests/Edit.spec.tsx:151-172, 258-272, apps/scratch/src/pages/tests/Solve.spec.tsx:140-165, 303-333
Test Execution and Configuration
Tests are executed via npm scripts defined in frontend/package.json. The main configuration for E2E tests resides in e2e/playwright.config.ts.
Key Configuration
| Configuration | Description | Source File |
|---|---|---|
webServer | Defines the command to start the test server, which includes seeding the database and launching the frontend, backend, and a mock OIDC provider. | e2e/playwright.config.ts:80-112 |
projects | Configures different test runs. Includes setup projects (authentication, setupFinish) that must run before the main test suite. | e2e/playwright.config.ts:41-78 |
workerConfig fixture | A custom fixture that sets up the isolated environment (database, servers) for each test worker. | e2e/helpers.ts:81-224 |
_databaseSeed fixture | A custom fixture that resets the worker's database to the clean template state whenever the test file changes, ensuring file-level test isolation. | e2e/helpers.ts:241-289 |
timeout | Sets a generous timeout of 120 seconds per test to accommodate the full-stack environment. | e2e/playwright.config.ts:75-76 |
Sources: e2e/playwright.config.ts, e2e/helpers.ts
Summary
The testing strategy for collimator is robust and developer-friendly. By leveraging Playwright for both E2E and component testing, it achieves broad coverage across the application stack. The standout feature is the E2E test setup, which provides fully isolated environments for each worker, drastically improving test reliability and enabling parallel execution. The use of established patterns like the Page Object Model and helper functions makes the test suite scalable and maintainable. Finally, the detailed component tests for the complex Scratch editor ensure that this critical, user-facing feature is thoroughly vetted.