Troubleshooting Guide
Relevant source files
The following files were used as context for generating this wiki page:
This guide provides comprehensive instructions for developers to diagnose, debug, and resolve issues within the FluentDo Agent project. It covers common problems related to building, testing, performance, and synchronizing with the upstream Fluent Bit repository. The agent is a hardened distribution of Fluent Bit, and troubleshooting often involves understanding its unique build process, dependencies, and patch management system.
For information on the release lifecycle and support, refer to the Support & Lifecycle section. For reporting security issues, please follow the guidelines in the main README.
General Troubleshooting
High-level troubleshooting starts with ensuring a correct build and runtime environment.
Building from Source
If you suspect a build-related issue, compiling the agent natively is a good first step. This requires all relevant dependencies to be installed on your system.
git clone https://github.com/FluentDo/agent.git
cd agent/source/build
cmake ..
make
Sources: README.md:104-109
For more complex build issues, especially cross-compilation, refer to the container-based build scripts.
Sources: README.md:95-99
Running with Docker
Using the official Docker images is a quick way to validate if an issue is environment-specific. You can run the agent and mount a local configuration file to test.
# UBI-based image
docker run --rm -it -v /var/log/containers:/var/log/containers:ro ghcr.io/fluentdo/agent/ubi:main -c /fluent-bit/etc/fluent-bit.yaml
# Debian-based image
docker run --rm -it -v /var/log/containers:/var/log/containers:ro ghcr.io/fluentdo/agent/debian:main -c /fluent-bit/etc/fluent-bit.yaml
Sources: README.md:83-86
Reporting Issues
For bugs and feature requests, use the official GitHub Issues page. For security vulnerabilities, email security@fluent.do directly.
Sources: README.md:126-127, README.md:133-136
Build and Compilation Issues
Troubleshooting build problems often involves enabling debug flags and understanding the project's dependencies.
CMake Build Options
The project and its core dependencies, like the Monkey web server, provide several CMake options to aid in debugging. These can be passed to the cmake command.
| Option | Component | Description |
|---|---|---|
MK_DEBUG | Monkey | Build with debug symbols. |
MK_BACKTRACE | Monkey | Enable backtrace feature for crash analysis. |
MK_VALGRIND | Monkey | Enable Valgrind support for memory debugging. |
YYJSON_BUILD_TESTS | yyjson | Build all tests for the yyjson library. |
YYJSON_ENABLE_SANITIZE | yyjson | Enable sanitizers (e.g., AddressSanitizer) for tests. |
YYJSON_ENABLE_VALGRIND | yyjson | Enable Valgrind memory checker for yyjson tests. |
YYJSON_ENABLE_COVERAGE | yyjson | Enable code coverage for yyjson tests. |
Sources: source/lib/monkey/CMakeLists.txt:23-41, source/lib/yyjson-0.12.0/doc/doxygen/html/building-and-testing.html
To use these options, for example, to enable debug symbols and Valgrind support for Monkey, you would run:
cmake .. -DMK_DEBUG=Yes -DMK_VALGRIND=Yes
Building with Different Compilers
To rule out compiler-specific issues, you can generate projects for different toolchains.
# Clang for Linux/Unix
cmake .. -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++
# Microsoft Visual Studio for Windows
cmake .. -G "Visual Studio 16 2019" -A x64
# Xcode for macOS
cmake .. -G Xcode
Sources: source/lib/yyjson-0.12.0/doc/doxygen/html/building-and-testing.html
Testing and Validation
The project relies on a suite of testing and analysis tools to ensure code quality and stability. Running these locally is a key part of troubleshooting.
This diagram shows the general workflow for testing a change.
Running the Test Suite
The test suite can be executed using ctest. This is essential for verifying that changes have not introduced regressions.
# First, build with tests enabled
cmake -E make_directory build; cd build
cmake .. -DYYJSON_BUILD_TESTS=ON
# Build the project
cmake --build .
# Run the tests
ctest --output-on-failure
Sources: source/lib/yyjson-0.12.0/doc/doxygen/html/building-and-testing.html
Memory Safety Validation
Memory-related bugs can be diagnosed using Valgrind or compiler sanitizers.
Valgrind
To run tests with the Valgrind memory checker:
cmake .. -DYYJSON_BUILD_TESTS=ON -DYYJSON_ENABLE_VALGRIND=ON
cmake --build .
ctest --output-on-failure
Sources: source/lib/yyjson-0.12-0/doc/doxygen/html/building-and-testing.html
Sanitizers
To build with sanitizers (e.g., AddressSanitizer, UndefinedBehaviorSanitizer) using GCC or Clang:
cmake .. -DYYJSON_BUILD_TESTS=ON -DYYJSON_ENABLE_SANITIZE=ON
cmake --build .
ctest --output-on-failure
Sources: source/lib/yyjson-0.12.0/doc/doxygen/html/building-and-testing.html
Static Analysis
Static analysis helps find bugs without executing the code. The project uses scan-build from Clang.
# Run static analysis on the build
scan-build make
Sources: source/lib/zstd-1.5.7/CONTRIBUTING.md
Code Style and Formatting
To avoid issues related to code style, you can use the provided tools to check and fix formatting. This is especially important when preparing contributions. The project uses clang-format for C/C++ and flake8/autopep8 for Python.
# Check for style errors in changed files
make style-check-changed
# Automatically fix style errors
make style-fix-changed
Sources: source/lib/librdkafka-2.10.1/CONTRIBUTING.md
Upstream Synchronization Issues
FluentDo Agent is a downstream distribution of Fluent Bit. The sync-upstream-patches.sh script is used to apply commits from the upstream repository. Troubleshooting this process is critical for keeping the agent up-to-date.
Patch Syncing Workflow
The following diagram illustrates the process of syncing patches from the upstream Fluent Bit repository.
Sources: scripts/sync-upstream-patches.sh
Patch Categories
The sync script categorizes patches to help decide which ones to apply. By default, only TECHNICAL and TESTS are recommended for application.
| Category | Color | Description |
|---|---|---|
TECHNICAL | Green | Bug fixes, security fixes, and core improvements. |
TESTS | Cyan | Additions or modifications to the test suite. |
PACKAGING | Yellow | Changes related to packaging (RPM, DEB) and distribution. |
VERSION | Blue | Commits that only bump the version number. |
WORKFLOWS | Red | CI/CD workflow changes (e.g., GitHub Actions). |
Sources: scripts/sync-upstream-patches.sh:118-124, scripts/sync-upstream-patches.sh:176-194
You can skip categories using the --skip flag:
./scripts/sync-upstream-patches.sh --from v4.0.10 --to v4.0.11 --skip PACKAGING,VERSION
Sources: scripts/sync-upstream-patches.sh:73-76
Troubleshooting Failed Patches
The script attempts to apply patches using several methods. If a patch fails, it is logged for manual resolution.
patch -p1: The primary method.patch -p1 --fuzz=2: A fallback with fuzz factor to handle minor differences.git apply: A last resort that sometimes works whenpatchfails.
If a patch fails, the script will log it to /tmp/fluent-bit-patches/failed.log and discard any partial changes to keep the working directory clean. You will need to manually inspect the .patch file and the corresponding source files to resolve the conflict.
Sources: scripts/sync-upstream-patches.sh:233-294
Creating a Sync Pull Request
After successfully applying and committing patches, the create-sync-pr.sh script automates the creation of a GitHub pull request.
The process is as follows:
- Checks for prerequisites like the
ghCLI and a clean working directory. - Collects commit information to generate a detailed PR body.
- Pushes the local branch to the remote.
- Creates the pull request with a standardized title and body.
Sources: scripts/create-sync-pr.sh
Performance Benchmarking
For performance-related issues, it's important to establish a stable benchmarking environment to get reliable results.
Creating a Stable Environment
- Disable frequency scaling and turbo boost to prevent CPU speed fluctuations.
- Reserve a CPU core for the benchmark process to avoid interference from other OS and user processes.
- Avoid storage I/O by using an in-memory filesystem like
tmpfsfor the program, inputs, and outputs. - Run benchmarks multiple times and aggregate the results (e.g., using the median or the fastest run) to reduce noise.
Sources: source/lib/zstd-1.5.7/CONTRIBUTING.md
Using Profilers
On Linux, perf is a powerful tool for profiling.
# Get quick timing and counter statistics
perf stat -r 10 <your-benchmark-program>
This command runs the program 10 times and provides statistics on CPU cycles, instructions, cache misses, and more, which can help pinpoint performance bottlenecks.
Sources: source/lib/zstd-1.5.7/CONTRIBUTING.md