Performance Benchmarking
Relevant source files
The following files were used as context for generating this wiki page:
Performance benchmarking in the Jurassic project is a critical process for measuring and ensuring the efficiency of the JavaScript engine. It is accomplished through two primary mechanisms: a standalone Benchmarker console application and a Benchmark utility method integrated into the unit testing framework. These tools execute standardized JavaScript benchmark suites, such as SunSpider and the V8 benchmark suite, to gather performance metrics. This allows developers to track performance regressions or improvements over time.
The Benchmarker application is designed for running a collection of script files and reporting aggregate statistics, while the TestBase.Benchmark method provides a more granular way to measure the performance of specific C# code blocks within individual unit tests.
Benchmarker Console Application
The Benchmarker is a dedicated console application for running a suite of JavaScript performance tests and summarizing the results. Its primary purpose is to execute the SunSpider benchmark suite.
Sources: Benchmarker/Program.cs
Execution Flow
The application's Main method orchestrates the benchmarking process. It first increases the process's thread priority to minimize interference from other system processes. It then proceeds to run the benchmark files, collecting and summarizing the timing data.
The following diagram illustrates the execution flow of the Benchmarker application.
Sources: Benchmarker/Program.cs:88-132
Core Logic
- Initialization: The process thread priority is set to
AboveNormalto ensure more consistent measurements.
Sources: Benchmarker/Program.cs:91 - Test Phases: The entire suite of tests is run 5 times. The first run acts as a warm-up, and its results are not included in the final statistical analysis.
Sources: Benchmarker/Program.cs:104, Benchmarker/Program.cs:128-130 - Test Execution: For each JavaScript file in the
sunspider-0.9.1directory, a newScriptEngineinstance is created. The script is read into memory, and its execution is timed usingSystem.Diagnostics.Stopwatch.
Sources: Benchmarker/Program.cs:113-123 - Data Collection: The elapsed time for each test run (after the first warm-up phase) is collected into a
Seriesobject. This class is a simple helper for calculating the mean and standard deviation of the collected samples.
Sources: Benchmarker/Program.cs:128-130, Benchmarker/Program.cs:11-40
Series Helper Class
This internal class is used to aggregate timing samples and calculate basic statistics.
Sources: Benchmarker/Program.cs:11-40
AddSample(double value): Adds a new timing measurement to the internal list.Mean: Calculates the average of all collected samples.StandardDeviation: Calculates the standard deviation of the samples.
Output
After all phases are complete, the application prints a summary to the console. For each test file, it displays the mean execution time and the standard deviation. Finally, it prints the total elapsed time for the mean of all tests.
Sources: Benchmarker/Program.cs:135-146
TestBase.Benchmark Utility
The TestBase class, a base for all unit tests, contains a static Benchmark method. This utility is designed to measure the performance of a specific piece of C# code, typically one that involves executing JavaScript, and report the results within the Visual Studio Test Explorer.
Sources: Unit Tests/Utils/TestBase.cs:115-189
Execution Flow
The Benchmark method is more sophisticated than the Benchmarker application. It is designed for micro-benchmarking within a unit test context.
The following diagram shows the logic of the Benchmark method.
Sources: Unit Tests/Utils/TestBase.cs:115-189
Core Logic
- High-Priority Execution: The thread priority is elevated to
ThreadPriority.Highestto get the most stable measurements possible.
Sources: Unit Tests/Utils/TestBase.cs:119 - Overhead Measurement: It measures the time taken to call an empty
Actionto calculate and later subtract the test harness overhead from each measurement.
Sources: Unit Tests/Utils/TestBase.cs:125-128 - Warm-up: A single, untimed run of the code is performed to ensure the code is JIT-compiled.
Sources: Unit Tests/Utils/TestBase.cs:131 - Timed Loop: The code is executed repeatedly for a fixed duration of 2 seconds. This is different from the
Benchmarker's fixed number of iterations.
Sources: Unit Tests/Utils/TestBase.cs:134-150 - Garbage Collection:
System.GC.Collect()is explicitly called after each timed iteration to prevent GC pauses from interfering with the measurements.
Sources: Unit Tests/Utils/TestBase.cs:147 - Results Reporting: After the loop, it calculates statistics and throws an
AssertInconclusiveException. This makes the result visible in the unit test output, formatted as "operations/sec".
Sources: Unit Tests/Utils/TestBase.cs:165
Comparison of Benchmarking Tools
The two main benchmarking tools serve different but related purposes.
| Feature | Benchmarker Application | TestBase.Benchmark Utility |
|---|---|---|
| Context | Standalone console application for batch testing. | Helper method for micro-benchmarking within unit tests. |
| Granularity | Entire script file execution. | Arbitrary C# Action delegate. |
| Iteration | Fixed number of runs (5 total, 4 measured). | Runs for a fixed duration (2 seconds). |
| Warm-up | First run's results are discarded. | An explicit, untimed warm-up run is performed. |
| GC Control | No explicit garbage collection. | GC.Collect() is called between each measured run. |
| Output | Console output (ms ± stdev). | AssertInconclusiveException (ops/sec) for test runners. |
| Source | Benchmarker/Program.cs | Unit Tests/Utils/TestBase.cs |
Benchmark Suites
The project utilizes standard, well-known JavaScript benchmark suites to test the engine's performance.
| Suite | Description | Example Files |
|---|---|---|
| SunSpider | A JavaScript benchmark from Apple, focused on real-world use cases. The Benchmarker application is configured to run files from the sunspider-0.9.1 directory. | 3d-cube.js |
| V8 Benchmark Suite | A collection of pure JavaScript benchmarks originally used to tune the V8 engine. It includes tests for ray tracing, cryptography, and other computationally intensive tasks. | raytrace.js, earley-boyer.js |
Sources: Benchmarker/Program.cs:103, Benchmarker/Files/v8/README.txt
The V8 benchmark suite has evolved over several versions to include fixes and new benchmarks, such as RegExp and Splay, to avoid incorrect optimizations and exercise the engine's memory management subsystem.