Filter Plugin: Lua

Relevant source files

The following files were used as context for generating this wiki page:

• source/plugins/filter_lua/lua.c
• source/src/flb_lua.c
• source/plugins/CMakeLists.txt
• source/src/fluent-bit.c
• source/lib/luajit-04dca791/doc/install.html

The Lua filter plugin provides a flexible way to modify log records on the fly using custom Lua scripts. It leverages the high-performance LuaJIT runtime to execute user-defined functions that can inspect, alter, or even drop records as they pass through the processing pipeline.

This plugin acts as a bridge between the Fluent Bit core and a Lua scripting environment. For each incoming record, it converts the data into a Lua-compatible table, invokes a specified Lua function, and then processes the function's return values to determine the final state of the record. This allows for complex data manipulation that goes beyond the capabilities of standard filters.

Architecture and Lifecycle

The plugin is built upon the LuaJIT library, which is a Just-In-Time compiler for the Lua 5.1 language. The integration is managed through a shared Lua context defined in flb_lua.c and the specific filter implementation in filter_lua/lua.c.

Plugin Initialization (cb_lua_init)

When the filter is initialized, it performs the following steps:

1. Reads configuration parameters such as script and call.
2. Creates a new Lua scripting context using flb_lua_init(). This function sets up a new Lua state (luaL_newstate).
3. Loads the specified Lua script file into the Lua state.
4. Verifies that the specified function (from the call parameter) exists within the loaded script.
5. Stores the script context for use during data processing.

The initialization flow is depicted below.

This diagram illustrates the setup process for the Lua filter plugin upon startup.
Sources: source/plugins/filter_lua/lua.c:198-254, source/src/flb_lua.c:60-76, source/lib/luajit-04dca791/doc/install.html

Plugin Registration

The Lua filter is registered within the build system conditionally. It is only included if the FLB_LUAJIT option is enabled during compilation.

Sources: source/plugins/CMakeLists.txt:385-387

if(FLB_LUAJIT)
  REGISTER_FILTER_PLUGIN("filter_lua")
endif()

Configuration

The Lua filter is configured within an [FILTER] section of the configuration file or via command-line arguments.

Sources: source/src/fluent-bit.c:605-611, source/plugins/filter_lua/lua.c:198-232

Parameter	Description	Default
script	Absolute path to the Lua script file that will be executed.	-
call	The name of the Lua function within the script that will be invoked for each record.	-
protected_mode	If enabled, the plugin runs in a protected mode where script errors do not halt the engine. If disabled, any script error will cause Fluent Bit to exit.	true
type_error	Specifies the action to take if the Lua script returns a data type that is not a table. Options are log (default) or drop.	log

Example Configuration

[FILTER]
    Name          lua
    Match         *
    script        /path/to/my_script.lua
    call          process_record

Data Processing Flow

The core logic resides in the cb_lua_filter function. For each chunk of records, the plugin iterates through them and performs the following actions for each record:

1. Data Conversion: The incoming msgpack record is converted into a Lua table using flb_lua_push_map. The timestamp is also pushed to the Lua stack.
2. Lua Function Call: The user-defined Lua function (specified by call) is invoked with three arguments: the record's tag, the timestamp, and the record data as a Lua table.
3. Process Return Values: The Lua function is expected to return three values: a status code, a new timestamp, and a new record (as a Lua table).
4. Handle Status Code:
   • 0 (FLB_LUA_KEEP): The record is kept. If the returned record is modified, it's converted back to msgpack using flb_lua_tomap.
   • 1 (FLB_LUA_DROP): The record is dropped and will not be forwarded.
   • 2 (FLB_LUA_MODIFIED): Deprecated, but treated the same as FLB_LUA_KEEP.
5. Update Record: If the record is kept, the new (or original) record is appended to the output buffer.

The following sequence diagram shows the interaction between the Fluent Bit engine and the Lua script.

Lua Scripting API

The user-defined Lua function must adhere to a specific signature and return value structure.

Function Signature

The function specified in the call configuration property receives three arguments:

1. tag (string): The tag associated with the record.
2. timestamp (number or table): The record's timestamp. It can be a floating-point number representing seconds since the epoch or a table with sec and nsec fields.
3. record (table): A Lua table representing the log record, where keys are strings and values are the corresponding record fields.

Return Values

The function must return three values:

1. code (number): An integer indicating the action to take.
   • 0 (FLB_LUA_KEEP): Keep the record. The record can be modified by returning a new table.
   • 1 (FLB_LUA_DROP): Drop the record.
2. new_timestamp (number or table): The new timestamp for the record. If the original timestamp should be kept, return the original timestamp argument.
3. new_record (table): The new record content. If the record was not modified, return the original record argument.

Example Script

This script adds a new key processed to each record.

-- my_script.lua
function process_record(tag, timestamp, record)
    -- Add a new field to the record
    record['processed_by_lua'] = true

    -- Keep the record, original timestamp, and return the modified record
    return 0, timestamp, record
end

Sources: source/plugins/filter_lua/lua.c:101-124

Conclusion

The Lua filter plugin is a powerful component within FluentDo Agent that enables advanced, stateful, and conditional log processing through custom scripts. By exposing records as native Lua tables and processing the script's return values, it offers a high degree of flexibility for filtering, enrichment, and routing tasks directly within the data pipeline, powered by the performance of LuaJIT.