Testing Strategy

Relevant source files

The following files were used as context for generating this wiki page:

• testing/bats/README.md
• source/tests/lib/shunit2/README.md
• source/lib/ctraces/tests/lib/acutest/acutest.h
• source/lib/yyjson-0.12.0/doc/doxygen/html/building-and-testing.html
• source/lib/librdkafka-2.10.1/CONTRIBUTING.md
• source/lib/zstd-1.5.7/CONTRIBUTING.md
• README.md

The FluentDo Agent project employs a multi-layered and comprehensive testing strategy to ensure its stability, security, and performance, aligning with its goal of being a "hardened, stable, secure by default" distribution. The strategy encompasses C/C++ unit tests, extensive shell-based functional and integration tests, and rigorous testing of third-party dependencies. This ensures "battle-tested quality" through continuous validation, memory safety checks, and performance benchmarking.

The overall testing approach can be visualized as a pyramid, with a foundation of unit tests, a middle layer of functional tests, and a top layer of complex integration tests.

This diagram illustrates the hierarchical testing structure, from low-level unit tests to high-level integration tests, and the primary frameworks used at each layer.

Sources: README.md:80-84, testing/bats/README.md:10-15, source/lib/ctraces/tests/lib/acutest/acutest.h

C/C++ Unit Testing with Acutest

The core C/C++ components of the agent and its internal libraries (like ctraces, cprofiles, and cmetrics) are tested using Acutest, a lightweight unit testing framework. This framework allows for detailed, low-level validation of individual functions and modules.

Sources: source/lib/ctraces/tests/lib/acutest/acutest.h, source/lib/cprofiles/tests/lib/acutest/acutest.h, source/lib/cmetrics/tests/lib/acutest/acutest.h

Key Features

• Test Suites: Tests are organized into suites using the TEST_LIST macro, which defines a list of test names and their corresponding function pointers.
• Assertions: The framework provides TEST_CHECK for standard assertions and TEST_ASSERT for critical assertions that abort the test on failure.
• Test Cases: Complex tests can be broken down into named sub-sections using TEST_CASE, which helps in pinpointing failures within looped or multi-stage tests.
• Process Isolation: By default, Acutest runs each test in a separate child process (on supported platforms like Unix and Windows) to prevent a crash in one test from halting the entire suite. This can be disabled with the --no-exec flag, which is useful for debugging.
• Output Formatting: Acutest supports various verbosity levels, colorized output, and can generate TAP (Test Anything Protocol) and XUnit XML reports for CI/CD integration.

Sources: source/lib/ctraces/tests/lib/acutest/acutest.h:35-47, 68-91, 130-145, 915-926, 1003-1011

Test Execution Flow

Acutest provides a main() entry point that handles command-line parsing, test discovery, and execution. The typical flow for an isolated test run is as follows:

This diagram shows how the main Acutest process forks a child for each test, waits for its exit code, and then aggregates the results into a final summary.

Sources: source/lib/ctraces/tests/lib/acutest/acutest.h:816-896, 1261-1335

Command-Line Options

Acutest provides a rich set of command-line options to control test execution.

Option	Alias	Description
--skip	-s	Execute all tests except the ones listed.
--exec[=WHEN]		Execute tests as child processes. WHEN can be auto, always, never.
--no-exec	-E	Disables child process execution for tests.
--time	-t	Measures and reports the duration of each test.
--tap		Produces TAP-compliant output for integration with other tools.
--xml-output=FILE	-x	Generates an XUnit XML report to the specified file.
--list	-l	Lists all available unit tests and exits.
--verbose[=LEVEL]	-v	Sets the verbosity level (0-3).
--color[=WHEN]		Controls colorized output (auto, always, never).
--help	-h	Displays the help message.

Sources: source/lib/ctraces/tests/lib/acutest/acutest.h:1013-1036

Shell Script Testing

The project utilizes two distinct shell script testing frameworks, BATS and shUnit2, to cover functional and integration scenarios, primarily focusing on the command-line interface (CLI), configuration, and interactions with external systems.

BATS (Bash Automated Testing System)

BATS is used for modern, higher-level functional and integration testing. The tests are structured to be reusable across different environments like containers, Kubernetes, and various operating systems.

Sources: testing/bats/README.md

Test Structure

The BATS tests are organized into two main categories:

1. Functional Tests (./tests/functional/): These are standalone tests that can be run with just the agent binary. They do not require external dependencies and are designed to be simple and portable.
2. Integration Tests (./tests/integration/): These are more complex tests that verify behavior with external infrastructure, such as sending data to a specific backend. They often require setting up additional services.

Helper functions and libraries are provided under helpers/ and lib/ to promote code reuse and simplify test creation.

This diagram shows the BATS test organization, where the run script uses tags to select and execute either functional or integration tests, both of which leverage common helper libraries.

Sources: testing/bats/README.md:6-21

Tagging System

Tests are tagged to allow for selective execution based on the target environment or test type. This is a critical feature for running relevant checks in different CI/CD pipelines.

Tag	Description
k8s	Tests specific to Kubernetes environments.
container	Tests designed to run inside a container.
linux	Linux-specific tests.
macos	macOS-specific tests.
windows	Windows-specific tests.
functional	Standalone tests without external dependencies.
integration	Tests requiring external infrastructure.

Tags can be applied to a whole file (# bats file_tags=...) or individual tests.

Sources: testing/bats/README.md:27-42

shUnit2

shUnit2 is an xUnit-style test framework used for testing Bourne-based shell scripts, ensuring high portability across various shells like sh, bash, dash, and ksh. It provides a structured way to write unit tests for shell functions and scripts.

Sources: source/tests/lib/shunit2/README.md:3-8, 40-47

Test Lifecycle

shUnit2 follows a classic xUnit lifecycle, automatically discovering and running test functions.

This diagram outlines the execution order of setup, test, and teardown functions in a shUnit2 test suite.

Sources: source/tests/lib/shunit2/README.md:63-84

Key Assertions

shUnit2 provides a comprehensive set of assertion functions to validate script behavior.

Function	Description
assertEquals	Asserts that two string or integer values are equal.
assertNotEquals	Asserts that two values are not equal.
assertNull	Asserts that a value is a zero-length string.
assertNotNull	Asserts that a value is not a zero-length string.
assertTrue	Asserts that a shell condition evaluates to true (exit code 0).
assertFalse	Asserts that a shell condition evaluates to false (exit code 1).

Sources: source/tests/lib/shunit2/README.md:109-160

Testing of Dependencies

The project's high quality standards extend to its third-party dependencies. The build and contribution guides for libraries like yyjson, librdkafka, and zstd reveal a strong emphasis on their own internal testing, which the FluentDo Agent project leverages.

• yyjson: Provides extensive CMake options to build and run tests (YYJSON_BUILD_TESTS), including support for Valgrind, sanitizers (ASan, UBSan), and code coverage analysis. This ensures the core JSON parser is robust and memory-safe.
• librdkafka: Mandates that all bug fixes and new features must be accompanied by new test cases in the regression suite. It uses clang-format for style enforcement and requires all tests to pass before a PR is considered.
• zstd: Has a multi-tiered testing approach (short, medium, long tests) that run on different CI triggers. It uses tools like ASan, UBSan, MSan, and Valgrind to check for memory errors and undefined behavior across multiple architectures.

This focus on dependency quality ensures that the foundation upon which the agent is built is itself stable and well-tested.

Sources: source/lib/yyjson-0.12.0/doc/doxygen/html/building-and-testing.html:1-1, 100-101, 138-141, source/lib/librdkafka-2.10.1/CONTRIBUTING.md:129-134, source/lib/zstd-1.5.7/TESTING.md:3-10, 16-21

Summary

The testing strategy for FluentDo Agent is robust and multi-faceted, reflecting the project's commitment to delivering a production-ready, high-quality product. By combining C/C++ unit tests for core logic, shell-based testing for functional and integration scenarios, and ensuring dependencies are rigorously validated, the strategy provides comprehensive coverage. The use of modern tools and practices like process isolation, test tagging, and integration with memory safety tools like Valgrind and AddressSanitizer contributes significantly to the agent's reliability and security posture.

Sources: README.md:80-84, source/lib/yyjson-0.12.0/doc/doxygen/html/building-and-testing.html:142-145