Developing Native C Modules
Relevant source files
The following files were used as context for generating this wiki page:
Native C modules in rsyslog extend the core engine's capabilities by providing logic for input, output, parsing, and message modification. The project employs a microkernel architecture where the core runtime manages orchestration, threading, and queuing, while loadable plugins implement specific protocol or storage logic.
Developing a native module requires adhering to the v8 worker model, which dictates specific concurrency and locking rules to ensure stability in multi-threaded environments. Modules are dynamically loaded and must implement a standardized interface defined in the runtime headers.
Sources: doc/source/development/architecture.rst:12-23, DEVELOPING.md:4-10
Architectural Overview
The rsyslog architecture separates the core engine from its extensions. The core (runtime) provides essential services like resource management and configuration parsing, while plugins handle the data lifecycle.
Module Classifications
Modules are identified by a specific prefix in their filenames:
| Prefix | Type | Description |
|---|---|---|
| im | Input | Producers that listen for data (e.g., imtcp, imfile). |
| om | Output | Consumers that send logs to destinations (e.g., omfile, omelasticsearch). |
| pm | Parser | Decoders for raw message formats (e.g., pmrfc5424). |
| mm | Message Mod | Transformers that modify log content (e.g., mmjsonparse). |
| fm | Function | Extension functions for RainerScript. |
Sources: doc/source/development/architecture.rst:31-40, doc/source/historical/module_devel.rst:43-46
Data Flow Pipeline
The core engine routes data through a multi-threaded pipeline where modules act as endpoints or intermediate processors.
The diagram illustrates the flow from an Input Module through the core Runtime and finally to an Output Module.
Sources: doc/source/development/architecture.rst:43-62
Concurrency and The v8 Worker Model
Modern rsyslog modules must support the v8 worker model, which allows multiple worker threads to execute actions concurrently. Proper state management is critical to avoid race conditions.
State Management
State is divided into two primary categories:
- Per-Action Data (
pData): Shared across all workers of a specific action. This must be guarded by mutexes if mutable. - Worker Instance Data (
wrkrInstanceData_t): Thread-local data owned by exactly one worker. No locks are required for members stored here.
Locking Rules
- Serial Resources: Resources that are inherently serial (like a shared file handle in
pData) must be protected by apthread_mutex_t. - Worker Isolation: Never share a
wrkrInstanceData_t(WID) across threads. - External Threads: If a module uses a library that spawns its own threads, an
rwlockshould be used to define clear read/write paths between the rsyslog worker and the library callback.
Sources: DEVELOPING.md:4-22, runtime/AGENTS.md:41-45
Implementation Entry Points
Modules must implement specific functions to hook into the engine's lifecycle. These are typically generated or managed via module-template.h.
Key Lifecycle Functions
createWrkrInstance/freeWrkrInstance: Responsible for allocating and freeing the thread-localwrkrInstanceData_t.doAction: The primary execution entry point. It may be called concurrently by different workers.- Transactional API: Includes
beginTransaction,commitTransaction, androllbackTransactionfor high-performance batching. These follow the same concurrency rules asdoAction.
Sources: DEVELOPING.md:28-32, runtime/AGENTS.md:14-16
Metadata and Maintenance
Every module must include a MODULE_METADATA.yaml file to communicate ownership and stability expectations.
Metadata Requirements
| Key | Values |
|---|---|
| support_status | core-supported, contributor-supported, stalled |
| maturity_level | fully-mature, mature, fresh, experimental, deprecated |
| primary_contact | Maintainer name or GitHub Discussion link |
| last_reviewed | Date in YYYY-MM-DD format |
Sources: plugins/AGENTS.md:28-63, plugins/MODULE_METADATA_TEMPLATE.yaml:1-8
Coding Standards
The project enforces strict coding and commenting styles to facilitate both human and AI contribution.
Doxygen Documentation
Mandatory Doxygen comments are required for:
- Public API Functions: Any function in a
.hfile. - File Headers: The
@fileblock at the top of.cand.hfiles. - Complex Logic: Any non-trivial implementation, even if static.
- Macros: Preprocessor macros should use
@brieftags.
Sources: COMMENTING_STYLE.md:10-25, COMMENTING_STYLE.md:47-56
Technical Requirements
- Modules are loaded directly into the address space and possess a high level of trust.
- New configuration parameters must be documented in
doc/source/reference/parameters/. - All module-specific tests should be placed in
tests/and named after the module (e.g.,omxyz-basic.sh).
Sources: doc/source/historical/module_devel.rst:54-62, plugins/AGENTS.md:86-90, plugins/AGENTS.md:95-97
Summary
Developing native C modules for rsyslog involves implementing specialized logic within a microkernel framework. Success depends on adhering to the v8 worker model's concurrency requirements, using the standardized lifecycle entry points, and maintaining comprehensive metadata. By following the project's architectural patterns and locking rules, developers can create high-performance, stable extensions for log processing.