Project Introduction
Relevant source files
The following files were used as context for generating this wiki page:
Rsyslog is a high-performance, modular logging framework designed for processing log pipelines in both traditional syslog environments and modern observability stacks. It supports a wide range of capabilities, including high-speed local message delivery, flexible routing, advanced filtering using RainerScript, and integration with external systems like Elasticsearch and Kafka.
The system is built on a modular microkernel-like architecture, separating core engine services from the specific logic required to ingest, parse, and output data. This design allows rsyslog to scale from small embedded environments to large enterprise deployments capable of processing over one million messages per second.
Sources: README.md:1-12, doc/source/index.rst:17-29, doc/source/development/architecture.rst:15-18
Core Architecture
Rsyslog follows a microkernel pattern where the core system provides essential infrastructure while business logic is contained in loadable modules.
The Core Engine (Runtime)
The core, located in the runtime/ directory, manages system resources, orchestration, and plugin interfaces. It is responsible for:
- Resource Management: Handling worker threads (
wtp.c), memory allocation, and internal queues. - Orchestration: Loading dynamically linked modules, parsing configuration files, and routing message objects.
- Interface Definitions: Providing the
modules.hcontract that all plugins must implement to interact with the engine.
Plugin Ecosystem
Functionality is extended through modules, categorized by their specific role in the pipeline:
| Module Prefix | Type | Description |
|---|---|---|
| im | Input | Producers that ingest logs (e.g., imtcp, imfile). |
| om | Output | Consumers that send logs (e.g., omelasticsearch, omfile). |
| pm | Parser | Decoders for raw message formats (e.g., pmrfc5424). |
| mm | Message Mod | Transformers for data enrichment (e.g., mmjsonparse). |
| fm | Function | Extension functions for RainerScript. |
Sources: doc/source/development/architecture.rst:21-43, doc/source/development/architecture.rst:64-70
Data Flow Pipeline
The rsyslog engine operates as an event-driven, multi-threaded producer-consumer pipeline. Data flows from a source through various stages before reaching a final destination.
The diagram illustrates the standard transition of a log message from ingestion to output.
Pipeline Stages
- Ingestion: An Input Module (e.g.,
imtcp) runs a thread to listen for data, creates a Message Object (smsg_t), and submits it to the core engine. - Buffering: The
qqueue_tcomponent buffers messages in memory or on disk to handle spikes and ensure delivery reliability. - Parsing & Routing: Worker threads dequeue message batches. Parsers decode the raw data, and the Ruleset Engine evaluates filters to determine appropriate actions.
- Execution: If filters match, messages are passed to an Action Queue. The associated Output Module then executes the delivery via a standard interface (e.g.,
doAction).
Sources: doc/source/development/architecture.rst:45-62
Internal Object System
Rsyslog implements a custom object system in C to provide encapsulation and polymorphism, located in runtime/obj.h. This system relies heavily on specific macros to simulate class-like behavior.
Lifecycle Management
Internal objects follow a strict lifecycle pattern:
- Construct: Allocation of memory for the object instance.
- SetProperty: Configuration of the instance's parameters.
- ConstructFinalize: Preparation of the instance for active use in the engine.
- Destruct: Teardown of the object and freeing of resources.
Sources: doc/source/development/architecture.rst:72-82
Documentation and Strategy
The project utilizes a "Docs-as-Code" strategy, where documentation is version-controlled and integrated into the CI/CD pipeline.
Implementation Principles
The documentation framework is built on four pillars:
- Automated Foundation: Reference documentation is derived directly from C source code using Doxygen, Breathe, and Sphinx to ensure accuracy.
- Narrative Layer: Human-authored guides and a library of validated configuration examples located in
doc/examples/. - Quality Gates: Continuous Integration (CI) checks for broken links, syntax warnings, and "drift" between code parameters and documentation.
- Ecosystem Integration: Support for man page generation and cross-referencing between related projects like
librelp.
Sources: doc/STRATEGY.md:18-70, doc/IMPLEMENTATION_PLAN.md:12-35, doc/source/development/doc_style_guide.rst:11-25
Summary
Rsyslog is a versatile, high-performance tool defined by its modularity and robust pipeline architecture. By separating the core engine from specialized plugins, it provides a stable platform for complex log processing and delivery. The project's commitment to "Docs-as-Code" and automated validation ensures that its technical reference remains authoritative for both human administrators and automated systems.