Send Queue Management

Relevant source files

The following files were used as context for generating this wiki page:

• src/sendq.c
• src/sendq.h
• src/sendbuf.c
• src/sendbuf.h
• src/relpsess.h
• src/relp.h
• doc/relp.html

Send Queue Management is a core component of the Reliable Event Logging Protocol (RELP) implementation in librelp. Its primary purpose is to manage outbound data buffers, ensuring reliable delivery even when the underlying transport layer (TCP) is temporarily unavailable or congested. The system provides a mechanism to queue frames, handle partial writes, and maintain a list of unacknowledged messages for potential retransmission.

The management system operates at the session level, utilizing a doubly-linked list of send queue entries. This architecture allows the RELP engine to handle asynchronous communication by buffering commands and responses until they can be successfully transmitted over the network. Sources: src/sendq.c:42-47, src/sendq.h:42-50, doc/relp.html:15-25

Architecture and Data Structures

The send queue is built upon two primary structures: the relpSendq object, which acts as the queue manager, and relpSendqe, which represents individual entries within the queue.

Core Components

• relpSendq_t: The root object for the send queue. It maintains pointers to the first (pRoot) and last (pLast) entries and includes a mutex (mut) for thread-safe operations. Sources: src/sendq.h:54-60
• relpSendqe_t: A doubly-linked list entry containing a reference to a relpSendbuf_t object. Sources: src/sendq.h:42-50
• relpSendbuf_t: The actual data container. It holds the raw bytes to be sent, tracking the current buffer pointer (bufPtr) to support partial writes. Sources: src/sendbuf.c:100-110, src/sendbuf.h

Relationship Diagram

The following diagram illustrates the relationship between the session, the queue, and the individual data buffers.

Data Flow and Transmission Logic

The transmission process involves adding buffers to the queue and attempting to drain the queue through the network socket.

Adding Data to the Queue

When a new buffer is added via relpSendqAddBuf, the system constructs a new entry (relpSendqe), appends it to the doubly-linked list, and immediately attempts an opportunistic send. This direct attempt saves a round of select() or epoll() if the socket is currently writable. Sources: src/sendq.c:126-155

Queue Processing (Draining)

The relpSendqSend function iterates through the queue starting from the root. For each entry, it calls relpSendbufSend. If a buffer is fully sent, it is removed from the queue and destructed. If a write is partial or fails with EAGAIN, the loop breaks, preserving the remaining data for the next transmission attempt. Sources: src/sendq.c:186-218

Unacknowledged Message Tracking

For reliable delivery, RELP clients must track messages that have been sent but not yet acknowledged by the server.

Unacked List Management

When a client sends a message using relpSendbufSendAll with the bAddToUnacked flag enabled, the buffer is moved to the session's unacknowledged list (pUnackedLstRoot). This list is stored within the relpSess_t structure. Sources: src/sendbuf.c:166-180, src/relpsess.h:112-115

Transaction Verification

When a response (rsp) command is received from the server, the client uses relpSessGetUnacked to locate the matching buffer based on the transaction number (txnr). If the server returns a successful status (Code 200), the buffer is finally destructed. Sources: src/scrsp.c:100-115

Function	Scope	Description
relpSendqAddBuf	Internal	Adds a buffer to the queue and attempts transmission.
relpSendqSend	Internal	Drains the queue as much as TCP allows.
relpSessAddUnacked	Client	Moves a sent buffer to the unacked tracking list.
relpSessGetUnacked	Client	Retrieves a buffer from the unacked list to process a response.
relpSendqIsEmpty	Global	Utility to check if the queue has pending data.

Sources: src/sendq.c, src/relpsess.h, src/sendbuf.c

Thread Safety and Mutex Usage

Because librelp is designed for highly threaded environments (like rsyslog), the send queue implementation utilizes a dedicated mutex (mut) within the relpSendq_t structure. This lock is acquired during:

• Adding a buffer to the queue.
• Deleting a buffer from the queue.
• Checking if the queue is empty.

This ensures that concurrent threads (e.g., a worker thread sending a message and a receiver thread processing an error hint) do not corrupt the doubly-linked list. Sources: src/sendq.c:88-91, src/sendq.c:115-117, src/sendq.c:174-176

Summary

Send Queue Management in librelp provides the necessary abstraction to transition from high-level RELP commands to low-level TCP writes. By managing doubly-linked lists of buffers and tracking unacknowledged transactions, it ensures that no data is lost during network congestion or session timeouts. The architecture's support for partial writes and thread-safe queue operations makes it suitable for high-performance, reliable logging applications. Sources: doc/relp.html:20-35, src/sendq.c:1-40