Automated Evaluation Engine

Relevant source files

The following files were used as context for generating this wiki page:

• services/code_runner.py
• routes/submit_routes.py
• levels/level_manager.py
• utils/security.py
• README.md
• templates/arena.html

The Automated Evaluation Engine is the core component of the Debug Arena platform responsible for the secure execution, validation, and scoring of participant-submitted Python code. It enables real-time competitive debugging by comparing program output against predefined expected results within a controlled environment.

This engine facilitates multi-level progressive difficulty by fetching challenge data, enforcing execution limits, and calculating scores based on time efficiency and hint utilization. It acts as the bridge between the browser-based IDE and the persistent storage of team progress.

Sources: README.md:14-18, services/code_runner.py:5-38

Core Architecture and Execution Flow

The engine operates through a multi-stage pipeline that begins with a submission from the frontend and concludes with score persistence in the database.

Code Execution Pipeline

1. Security Validation: Before execution, code is scanned for forbidden modules and functions.
2. Subprocess Spawning: The code is executed in a separate Python process using subprocess.run.
3. Result Capture: Standard output (stdout) and errors (stderr) are captured and returned to the routing layer.
4. Comparison: The captured output is trimmed and compared against the expected_output defined in the challenge metadata.

Sources: services/code_runner.py:7-27, routes/submit_routes.py:35-37

The following diagram illustrates the sequence of events during a code submission:

Security and Sanity Enforcement

The evaluation engine employs a "Gatekeeper" philosophy to prevent malicious code from compromising the host system or creating denial-of-service conditions.

Security Constraints

• Module Blacklisting: Uses a regex-based scanner to block dangerous keywords such as os, sys, subprocess, eval, and exec.
• Resource Limits: Implements a hard timeout of 2 seconds for every execution to mitigate infinite loops.
• Payload Size: Code submissions exceeding 2,000 characters are automatically rejected.
• Shell Protection: Subprocesses are executed with shell=False to prevent shell injection attacks.

Sources: utils/security.py:4-18, services/code_runner.py:16-20

Security Feature	Implementation Detail	Threshold/Value
Execution Timeout	subprocess.run timeout parameter	2 Seconds
Max Code Length	Character count check	2000 Characters
Module Blocking	String membership check (Forbidden list)	Restricted keywords
Execution Context	sys.executable in isolated process	shell=False

Sources: services/code_runner.py:19, utils/security.py:5-16

Scoring and Progression Logic

The engine calculates points dynamically based on performance metrics stored in the session and database.

Scoring Formula

The final score for a level is determined by the following factors:

1. Base Points: A starting value of 100 points.
2. Time Penalty: 1 point is deducted for every 10 seconds elapsed since the level was opened.
3. Hint Penalty: If the "Oracle" system was used, a level-specific penalty (e.g., 10-30 points) is applied.
4. Hard Minimum: Scores are capped at a minimum of 30-50 points to reward completion regardless of speed.

Sources: routes/submit_routes.py:43-58, levels/level_manager.py:8-135

Level Definition and Validation

Challenges are managed through the LEVELS data structure, which provides the engine with the necessary metadata for evaluation.

Challenge Data Structure

Each level entry includes:

• buggy_code: The initial code provided to the user.
• expected_output: The literal string result required for a successful evaluation.
• penalty: The point deduction associated with hint usage for that specific challenge.

Sources: levels/level_manager.py:3-135

# Example of level definition used by the engine
LEVELS = {
    1: {
        "title": "The Missing Link",
        "expected_output": "Hello, Coder",
        "penalty": 15,
        "difficulty": "Easy"
    }
}

Sources: levels/level_manager.py:4-16

Conclusion

The Automated Evaluation Engine provides a robust and secure framework for real-time code assessment. By combining strict security protocols with a dynamic scoring system, it ensures that the competitive integrity of the Debug Arena is maintained while providing participants with immediate feedback on their debugging efforts.