Background Processing (Piscina)
Relevant source files
The following files were used as context for generating this wiki page:
The Collimator backend leverages the Piscina library to manage a worker thread pool for executing CPU-intensive tasks asynchronously. This approach prevents long-running operations from blocking the main Node.js event loop, ensuring the application remains responsive. The primary use case for this background processing is the conversion of student code submissions (solutions) into a generalized Abstract Syntax Tree (AST), which is a prerequisite for subsequent analysis.
The process is initiated when a student submits a solution. The backend receives the submission and dispatches a job to a Piscina worker thread. This worker parses the solution file and transforms it into a standard AST format. The completion of this job is asynchronous, and frontend components poll the API to retrieve the analysis results once they are available.
Architecture Overview
The background processing architecture is centered around the AstConversionService, which manages the Piscina worker pool. When a new solution requires analysis, a job is sent to this service, which then offloads the heavy computation to a separate thread.
The following diagram illustrates the high-level interaction between components during solution analysis.
Core Components
AstConversionService
This NestJS injectable service is the central component for managing AST conversion tasks. It encapsulates the Piscina worker pool and exposes a method to run conversions.
Worker Pool Initialization
The service initializes a Piscina instance, configuring it to use a specific worker script. The getPiscinaPath utility is used to resolve the correct path to the compiled JavaScript worker file.
// backend/src/ast/ast-conversion.service.ts:13-23
export class AstConversionService implements OnModuleDestroy {
private readonly conversionWorker = new Piscina<
Parameters<ConversionWorker>[0],
ReturnType<ConversionWorker>
>({
filename: getPiscinaPath(
resolve(
__dirname,
// use the .js extension because NestJS compiles the typescript files
"./converters/solution-conversion-worker.piscina.js",
),
),
});
Sources: backend/src/ast/ast-conversion.service.ts:13-23
The onModuleDestroy lifecycle hook is implemented to ensure the worker pool is gracefully destroyed when the application shuts down, preventing resource leaks.
// backend/src/ast/ast-conversion.service.ts:25-27
onModuleDestroy(): void {
this.conversionWorker.destroy();
}
Sources: backend/src/ast/ast-conversion.service.ts:25-27
convertSolutionToAst Method
This method is the public API for triggering a background conversion. It takes a task and a solution as input and returns a promise that resolves with the GeneralAst.
The method currently has specific logic for SCRATCH task types, throwing an error for unsupported combinations. The actual work is dispatched to the worker pool via this.conversionWorker.run().
// backend/src/ast/ast-conversion.service.ts:33-49
async convertSolutionToAst(
task: TaskWithoutData,
solution: Solution,
): Promise<GeneralAst> {
let ast: Promise<GeneralAst>;
if (
task.type === TaskType.SCRATCH &&
solution.mimeType === "application/json"
) {
ast = this.conversionWorker.run({
solution,
taskType: task.type,
});
} else {
throw new Error(
`Unsupported (task, solution mime type) tuple '(${task.type}, ${solution.mimeType})'`,
);
}
return ast;
}
Sources: backend/src/ast/ast-conversion.service.ts:33-49
The data passed to the worker thread is a simple object:
| Parameter | Type | Description |
|---|---|---|
solution | Solution | The solution database object. |
taskType | TaskType | The type of the task (e.g., SCRATCH). |
| Sources: backend/src/ast/ast-conversion.service.ts:43-44 |
solution-conversion-worker.piscina.ts
This file defines the actual function that executes in the worker thread. While its content is not fully provided in the context, its role and interface are defined by its usage in AstConversionService. It is a function that accepts the solution and taskType and performs the computationally expensive conversion to a GeneralAst.
Sources: backend/src/ast/ast-conversion.service.ts:10, 14-23
End-to-End Flow
The entire process, from submission to analysis consumption, is asynchronous.
This sequence diagram shows the detailed flow:
Triggering and Monitoring
-
Submission: A user submits a solution via the
solve.tsxpage, which calls thecreateSolutionhook, triggering a request to the backend.
Sources: frontend/src/pages/class/[classId]/session/[sessionId]/task/[taskId]/solve.tsx:112 -
Asynchronous Confirmation: The asynchronous nature of the analysis is confirmed by the end-to-end test in
session-analysis.spec.ts. The test submits solutions and then enters awaitUntilloop, repeatedly polling an API endpoint (/api/v0/solutions/analysis/current/{classId}/{sessionId}/{taskId}) until the number of returned analyses matches the number of submissions. This demonstrates that the analysis is not completed in the same transaction as the submission.// e2e/tests/sessions/session-analysis.spec.ts:80-111 await waitUntil( () => page.evaluate( async ({ url, expectedStudentAnalyses }) => { // ... fetch logic ... const analyses = (await response.json()) as CurrentAnalysesDto; return ( analyses.studentAnalyses.length === expectedStudentAnalyses ); }, { url, expectedStudentAnalyses: solutions.length }, ), 10, 500, ); -
Consuming Results: Frontend components, such as
DissimilarityAnalysis.tsx, use SWR hooks likeuseCurrentSessionTaskSolutionsto fetch the analysis results and display them to the user.
Sources: frontend/src/components/dashboard/DissimilarityAnalysis.tsx:55-60
Summary
The Collimator project effectively uses Piscina to offload the CPU-bound task of AST conversion to a background worker pool. This architectural choice is crucial for maintaining a responsive backend server, especially under load. The AstConversionService provides a clean abstraction over the worker pool, and the asynchronous flow is handled gracefully from submission to the presentation of results in the user dashboard.