Adding New Analysis Criteria

Relevant source files

The following files were used as context for generating this wiki page:

• frontend/src/data-analyzer/analyze-asts.ts
• frontend/src/components/dashboard/criteria/criterion-base.ts
• frontend/src/components/dashboard/criteria/statement/index.ts
• frontend/src/components/dashboard/Analyzer.tsx
• frontend/src/components/dashboard/Analysis.tsx
• frontend/src/components/dashboard/criteria/function-declaration/index.ts
• frontend/src/components/dashboard/criteria/test/index.ts
• frontend/src/task/TaskForm.tsx

Analysis Criteria are a core, extensible feature of the Collimator dashboard that allows educators to analyze, visualize, and filter student code submissions based on specific, quantifiable metrics. These criteria can be derived from Abstract Syntax Trees (ASTs) of the code or from metadata associated with the submission (like test results).

This document outlines the architecture of the criteria system and provides a step-by-step guide for developers to add new analysis criteria. The system is designed to be modular, with each criterion defined by a set of standard interfaces, making it straightforward to extend the platform's analytical capabilities.

Core Concepts

The criteria system is built around a few key interfaces and types that define the behavior of each criterion. These are primarily located in frontend/src/components/dashboard/criteria/criterion-base.ts.

Criterion Types

There are two main categories of criteria:

• AstCriterionType: Criteria based on static analysis of the code's Abstract Syntax Tree (AST). Examples include counting the number of statements, expressions, or function declarations.
• MetaCriterionType: Criteria based on metadata about the solution, such as the number of passing tests.

These are combined into a single CriterionType union.

Sources: frontend/src/data-analyzer/analyze-asts.ts:5-15, frontend/src/components/dashboard/criteria/test/index.ts:11

Core Interfaces

The following diagram shows the main interfaces that a new criterion must implement or use.

This diagram illustrates the base interfaces for defining analysis criteria.

Sources: frontend/src/components/dashboard/criteria/criterion-base.ts:6-55, frontend/src/components/dashboard/criteria/statement/index.ts:16-20

CriterionAxisDefinition

This interface defines how a criterion can be used as an axis (X or Y) on the analysis scatter plot.

Property	Type	Description
criterion	CriterionType	The unique identifier for the criterion.
messages	(taskType) => { name: MessageDescriptor }	A function returning internationalization (i18n) messages for the criterion's name.
config	AxisConfig	Chart.js configuration for the scale, such as type: 'linear' and tick precision.
getAxisValue	(analysis: CurrentAnalysis) => number	A function that takes a single solution analysis and returns the numerical value for the axis.

Sources: frontend/src/components/dashboard/criteria/criterion-base.ts:28-36

CriterionFilterDefinition

This interface defines how a criterion can be used to filter the set of analyzed solutions.

Property	Type	Description
criterion	CriterionType	The unique identifier for the criterion.
formComponent	CriterionFormComponent	The React component used to render the filter's configuration options in the UI.
messages	(taskType) => { name: MessageDescriptor }	A function returning i18n messages for the criterion's name.
initialValues	Criterion	An object containing the default values for the filter form.
run	(criterion, analyses) => { matchesFilter, parameters }	The core filtering logic. It receives the filter configuration and all analyses, and must return a boolean array (matchesFilter) indicating which analyses pass the filter, along with any calculated parameters.

Sources: frontend/src/components/dashboard/criteria/criterion-base.ts:46-55

Steps to Add a New AST-Based Criterion

The following steps use the statement criterion as an example.

1. Define the Criterion Type

Add your new criterion to the AstCriterionType enum.

File: frontend/src/data-analyzer/analyze-asts.ts

export enum AstCriterionType {
  condition = "condition",
  customFunctionCall = "customFunctionCall",
  expression = "expression",
  // ... other criteria
  statement = "statement",
  // Add your new criterion here
  // myNewCriterion = "myNewCriterion",
}

Sources: frontend/src/data-analyzer/analyze-asts.ts:5-15

2. Implement the Analysis Logic

The core analysis logic resides in the analyzeAst function, which uses a match statement to delegate to the appropriate analyzer method from CriteriaBasedAnalyzer.

The overall data flow for analysis is as follows:

This diagram shows how UI components trigger analysis logic, which ultimately calls analyzeAst.

To add your logic:

1. (Optional but recommended) Add a new static method to the CriteriaBasedAnalyzer class in frontend/src/data-analyzer/criteria-based-analyzers/index.ts. This method will contain the logic to traverse the AST and compute the metric.
2. Add a new .with() clause to the match statement in analyzeAst to call your new analyzer method.

File: frontend/src/data-analyzer/analyze-asts.ts

// @ts-expect-error Not sure how to fix the typescript issue
export const analyzeAst: AnalysisFunction = (ast, input) =>
  match(input)
    .with(
      { criterion: AstCriterionType.statement },
      ({ criterion, input }) => ({
        criterion,
        output: CriteriaBasedAnalyzer.countStatements(ast, input),
      }),
    )
    // ... other .with() clauses
    /*
    .with(
      { criterion: AstCriterionType.myNewCriterion },
      ({ criterion, input }) => ({
        criterion,
        output: CriteriaBasedAnalyzer.countMyNewCriterion(ast, input),
      }),
    )
    */
    .exhaustive();

Sources: frontend/src/data-analyzer/analyze-asts.ts:83-145

3. Create the Criterion Definition File

Create a new directory for your criterion (e.g., frontend/src/components/dashboard/criteria/my-new-criterion/) and add an index.ts file. This file will define the criterion's behavior for both axes and filters.

A concrete example is the Statement criterion.

File: frontend/src/components/dashboard/criteria/statement/index.ts

// 1. Define filter and parameter interfaces
export interface StatementFilterCriterion extends CriterionBase<Criterion> {
  minimumCount: number;
  maximumCount: number;
}

export interface StatementFilterCriterionParameters
  extends CriterionBase<Criterion> {
  minNumberOfStatements: number;
  maxNumberOfStatements: number;
}

// 2. Implement CriterionAxisDefinition
export const StatementCriterionAxis: CriterionAxisDefinition<Criterion> = {
  criterion,
  messages: () => messages,
  config: {
    type: "linear",
    ticks: { precision: 0 },
  },
  getAxisValue: (analysis) => {
    return analyzeAst(analysis.generalAst, {
      criterion,
      input: undefined,
    }).output;
  },
};

// 3. Implement CriterionFilterDefinition
export const StatementCriterionFilter: CriterionFilterDefinition<
  Criterion,
  StatementFilterCriterion,
  StatementFilterCriterionParameters
> = {
  criterion,
  formComponent: StatementCriterionFilterForm, // Link to the UI component
  messages: () => messages,
  initialValues: {
    criterion,
    minimumCount: 0,
    maximumCount: 100,
  },
  run: (config, analyses) => {
    const numberOfStatementsList = analyses.map(
      (analysis) =>
        analyzeAst(analysis.generalAst, toAnalysisInput(config)).output,
    );

    return {
      matchesFilter: numberOfStatementsList.map(
        (numberOfStatements) =>
          config.minimumCount <= numberOfStatements &&
          config.maximumCount >= numberOfStatements,
      ),
      parameters: {
        criterion,
        maxNumberOfStatements: Math.max(...numberOfStatementsList),
        minNumberOfStatements: Math.min(...numberOfStatementsList),
      },
    };
  },
};

Sources: frontend/src/components/dashboard/criteria/statement/index.ts:16-74

4. Create the Filter Form Component

Create a React component that will be used to configure the filter. This component receives value, parameters, and onChange as props.

For the statement criterion, this would be StatementCriterionFilterForm.tsx. It would typically contain input fields (e.g., for minimumCount and maximumCount) that call the onChange prop when their values change. This component is then imported and referenced in the formComponent property of the CriterionFilterDefinition.

Sources: frontend/src/components/dashboard/criteria/statement/index.ts:11, frontend/src/components/dashboard/criteria/criterion-base.ts:9-21

5. Integrate the New Criterion

Finally, import your new CriterionAxisDefinition and CriterionFilterDefinition into the central aggregators for axes and filters. This makes them available to the Analysis and AnalyzerFilterForm components, respectively.

• For axes, add the CriterionAxisDefinition to the array in frontend/src/components/dashboard/axes/index.ts.
• For filters, add the CriterionFilterDefinition to the array in frontend/src/components/dashboard/filter/index.ts.

Once integrated, the new criterion will automatically appear in the "x-Axis", "y-Axis", and "Filtering Criteria" dropdowns on the analysis dashboard.

Sources: frontend/src/components/dashboard/Analysis.tsx:608-614, frontend/src/components/dashboard/Analyzer.tsx:220-229

Summary

The analysis criteria system in Collimator is highly modular. By following the pattern of defining a type, implementing the core analysis logic, and creating axis and filter definitions based on a set of standard interfaces, developers can easily extend the platform's analytical capabilities. This structure ensures that new criteria are seamlessly integrated into the dashboard's visualization and filtering features.