Performance & Benchmarks

Relevant source files

The following files were used as context for generating this wiki page:

• documents/HttpPerformanceBenchmark.md
• benchmarks/http/scripts/run.sh
• benchmarks/http/scripts/build.sh
• benchmarks/http/scripts/report.py
• benchmarks/http/servers/hxlibs_server.cpp
• examples/benchmarks/01_server.cpp

The HXLibs performance benchmarking system evaluates the HTTP coroutine server's efficiency against various industry-standard implementations. It measures throughput (RPS), latency percentiles, and stability across different workloads including short responses, JSON APIs, and heavy file I/O. The system uses automated workflows and fixed environments to ensure results are comparable and transparent.

Sources: documents/HttpPerformanceBenchmark.md:3-5, benchmarks/http/scripts/report.py:7-13

Benchmarking Architecture

The system executes benchmarks by isolating the server and the load generator (wrk) on separate CPU sets to prevent resource contention. It employs a multi-round execution strategy where the server process starts once per implementation and optimization level, then sequentially processes all test scenarios.

Component Interaction Flow

The following diagram illustrates the lifecycle of a single benchmark execution, from environment setup to report generation.

Test Scenarios and Workloads

The benchmark covers eight distinct scenarios to simulate diverse production environments. These range from low-concurrency latency tests to high-concurrency mixed traffic.

Scenario	Path	Concurrency Level	Focus
Short Response	/	Low (Workers × 8)	Framework overhead and scheduling
JSON API	/api/users	Standard (Workers × 128)	Typical API response performance
Route & Query	/api/users/...	Standard	Dynamic routing and parameter parsing
HTML (Memory)	/page.html	Standard	In-memory static content delivery
HTML (Disk)	/page-file.html	Standard	Real file system I/O overhead
64 KiB (Memory)	/payload.bin	Low	Large payload memory copying
64 KiB (Disk)	/payload-file.bin	Low	File transfer via useRangeTransferFile
Mixed Traffic	Mixed Paths	High (Standard × 2)	Realistic routing and size distribution

Sources: documents/HttpPerformanceBenchmark.md:33-46, benchmarks/http/scripts/run.sh:195-206

Implementation Details

Server Configurations

Each implementation adheres to official best practices to ensure fair comparison. HXLibs specifically utilizes its coroutine-based I/O for both memory and disk variants.

• HXLibs Memory: Uses the unified memory response path.
• HXLibs Disk: Employs the useRangeTransferFile coroutine API for asynchronous file transmission.
• Asio: Pre-generates response bytes and enables TCP_NODELAY.
• nginx: Baseline reference using sendfile, tcp_nopush, and open_file_cache.

Sources: documents/HttpPerformanceBenchmark.md:16-28, benchmarks/http/servers/hxlibs_server.cpp:65-71

Data Collection and Metrics

The system records raw data for every round to identify performance drifts and outliers. The report.py script aggregates these into several key metrics:

• Throughput (RPS): Measured as the median requests per second across all repeats.
• Latency Percentiles: Full curves from P50 to P99.99, plus absolute Max.
• Stability (CV): The Coefficient of Variation indicates the consistency of the implementation.
• Throughput Gap: The percentage difference between a specific implementation and the best performer in the same scenario.

Sources: benchmarks/http/scripts/report.py:53-75, benchmarks/http/scripts/report.py:126-140

Build and Execution Process

The benchmarking suite requires a specific build sequence to prepare the environment and external dependencies.

Step 1: Binary Preparation

The build.sh script fetches source code for external libraries (Asio, yalantinglibs, nginx, wrk) and compiles them using Clang with -O3 -DNDEBUG optimizations.
Sources: benchmarks/http/scripts/build.sh:14-25

Step 2: Environment Setup

The runner script calculates CPU affinity based on the available hardware. It splits the CPUs between the server and the client to avoid context-switching interference.

Step 3: Warmup and Sampling

JIT-based runtimes (FastAPI and Spring Boot) receive an additional 20-second global warmup (BENCH_RUNTIME_WARMUP) to allow the runtime to converge before sampling begins. Each implementation then undergoes a per-scenario warmup followed by the formal sampling period.
Sources: documents/HttpPerformanceBenchmark.md:7-10, benchmarks/http/scripts/run.sh:220-225

Reporting and Transparency

The final output is an interactive HTML report. It includes:

1. Throughput Curves: Visualizes machine drift and outlier rounds.
2. Latency Curves: Logarithmic scale comparison of tail latencies.
3. Optimization Comparison: Shows the delta between O2 and O3 builds (defaulting to O3).
4. Hardware Metadata: Logs compiler versions, kernel info, and memory to ensure reproducibility.

Sources: documents/HttpPerformanceBenchmark.md:52-62, benchmarks/http/scripts/report.py:270-290

Conclusion

The HXLibs performance benchmarking framework provides a rigorous, automated method for validating high-performance C++ networking. By using best-practice configurations for all participants and isolating system resources, it offers a realistic baseline for evaluating coroutine-based HTTP server implementations.