Testing Strategy

Relevant source files

The following files were used as context for generating this wiki page:

• Test Suite Runner/Program.cs
• Unit Tests/Utils/TestBase.cs
• Test Suite Runner/Code/TestSuite.cs
• Test Suite Runner/Kangax/KangaxTestSuite.cs
• Benchmarker/Program.cs
• Test Suite Runner/Test Suite Files/harness/sth.js
• Test Suite Runner/Test Suite Files/harness/helper.js
• Unit Tests/Real-world/RealWorldTests.cs
• Unit Tests/Core/DocumentationSamples.cs

The Jurassic project employs a comprehensive, multi-layered testing strategy to ensure correctness, standards compliance, and performance. This strategy combines traditional unit tests, integration with external ECMAScript test suites, real-world application testing, and performance benchmarking. This robust approach validates the JavaScript engine from core functionality up to complex, practical applications.

The testing infrastructure is primarily divided into three categories:

1. Unit and Integration Tests: C#-based tests using MSTest for fine-grained validation of engine features.
2. External Suite Runner: A dedicated runner for executing large, external JavaScript test suites like Kangax and Test262 to ensure spec compliance.
3. Benchmarking: A tool for measuring and tracking the performance of the engine using standardized benchmarks like SunSpider.

Unit Testing Framework

Unit tests are built using the MSTest framework and are organized into logical namespaces like Core, Library, and Real-world. A base class, TestBase, provides common infrastructure and helper methods for all tests.

Sources: Unit Tests/Utils/TestBase.cs, Unit Tests/Core/DocumentationSamples.cs, Unit Tests/Real-world/RealWorldTests.cs

The TestBase Class

The TestBase class is the foundation for all unit tests. It provides a clean ScriptEngine instance for each test method, ensuring test isolation.

• Test Initialization: A new ScriptEngine is provisioned for each test via the [TestInitialize] method, preventing state leakage between tests.
• Helper Methods: It offers a suite of helper methods to simplify test authoring.

The following diagram shows how test classes inherit from TestBase to gain access to the testing infrastructure.

Key helper methods in TestBase include:

Method	Description
Evaluate(string script)	Evaluates a JavaScript snippet and returns the result.
Execute(string script)	Executes a JavaScript snippet without returning a result.
EvaluateExceptionType(string script)	Evaluates a script and returns the type of the JavaScript exception thrown.
EvaluateExceptionMessage(string script)	Evaluates a script and returns the message of the JavaScript exception thrown.
EvaluateAccessibility(string, string)	Checks the property attributes (enumerable, writable, configurable) of an object's property.
Benchmark(Action codeToTest)	Runs a piece of code multiple times to measure its performance.

Sources: Unit Tests/Utils/TestBase.cs:27-167

Real-World Application Testing

To ensure practical compatibility, Jurassic includes a suite of tests that run popular, real-world JavaScript libraries. This validates that the engine can handle complex, non-trivial code beyond simple spec-compliance tests.

Examples of tested libraries include:

• Showdown (Markdown processor)
• CoffeeScript (Compiler)
• RSA, MD5, SHA1, SHA256 (Cryptography)
• LZW (Compression)

A typical test involves loading the library's script file and then executing its functions to verify the output against expected results.

Sources: Unit Tests/Real-world/RealWorldTests.cs

External Test Suite Integration

A significant part of the testing strategy is the ability to run external, standardized test suites. This is crucial for verifying ECMAScript compliance. The project includes a TestSuiteRunner for this purpose, which supports suites like the Kangax compatibility table tests.

Sources: Test Suite Runner/Program.cs, Test Suite Runner/Code/TestSuite.cs, Test Suite Runner/Kangax/KangaxTestSuite.cs

C# Test Suite Runner

The C# runner is designed to execute tests in an isolated process, preventing a crash in the JavaScript engine from terminating the entire test run.

Architecture

The runner uses a parent-child process model with communication over named pipes.

1. TestSuite Base Class: An abstract class that defines the interface for a test suite, including Id, Name, and the Run() method. It also tracks statistics like success, failure, and skipped test counts.
2. KangaxTestSuite: A concrete implementation for running the Kangax ES6 compatibility tests. It reads test cases from a JSON file (compat-table.json).
3. Process Isolation: The main Program.cs starts a child process for the test suite. The KangaxTestSuite acts as a PipeServer in the main process, sending test scripts to the worker process and receiving results. This ensures that if a test crashes the engine, only the worker process is affected.

This diagram illustrates the class hierarchy for test suites.

Execution Flow

The sequence diagram below shows how the KangaxTestSuite runs a single test.

Browser-based Test Harness

The project also contains a complete JavaScript-based test harness for running suites like Test262 in a web browser. This harness dynamically loads tests, runs them in an <iframe>, and presents the results in a user-friendly interface.

Sources: Test Suite Runner/Test Suite Files/harness/sth.js, Test Suite Runner/Test Suite Files/harness/helper.js, Test Suite Runner/Test Suite Files/harness/sections.js

Components

The harness is composed of several key JavaScript classes:

Class	File	Description
Controller	sth.js	The main orchestrator. Manages starting, pausing, and resetting the test run.
TestLoader	sth.js	Loads test cases from JSON files and provides them to the BrowserRunner.
BrowserRunner	sth.js	Executes a single test case within an isolated <iframe> to catch errors and prevent script interference.
Presenter	helper.js	Manages the UI, including updating the progress bar, logging results, and rendering the final report.
Section	sections.js	Represents a section of the ECMAScript specification, used to group tests and aggregate results.

Sources: Test Suite Runner/Test Suite Files/harness/sth.js, Test Suite Runner/Test Suite Files/harness/helper.js, Test Suite Runner/Test Suite Files/harness/sections.js

Execution Flow

The following flowchart shows the lifecycle of the browser-based test harness.

Performance Testing

Performance is a key goal, and the Benchmarker console application is used to measure it. It runs the SunSpider benchmark suite to produce consistent and comparable performance metrics.

Sources: Benchmarker/Program.cs

Benchmarking Process

The process is straightforward:

1. The application identifies all .js files in the sunspider-0.9.1 directory.
2. For statistical accuracy, it runs each benchmark script multiple times (5 by default). The first run is often discarded to account for JIT warmup.
3. System.Diagnostics.Stopwatch is used for precise timing of each run.
4. A custom Series class collects the timing samples for each benchmark.
5. Finally, it calculates and prints the mean execution time and standard deviation for each benchmark, along with a total score.

This diagram outlines the benchmarking flow.