Data Buffering Strategy
Relevant source files
The following files were used as context for generating this wiki page:
The Data Buffering Strategy in librelp is designed to ensure the reliable delivery of event logs over the Reliable Event Logging Protocol (RELP). It employs a multi-layered approach to handle data during various stages of a transaction, from initial command construction to final acknowledgement. The strategy balances performance and reliability by using memory-resident queues for outgoing data and maintaining an "unacked" list to facilitate retransmission in case of connection failure.
At its core, the strategy relies on relpSendbuf objects to wrap raw protocol frames and relpSendq queues to manage the flow of data to the transport layer. This architecture supports full-duplex communication and command-response windows, allowing multiple commands to be outstanding simultaneously while ensuring they are acknowledged in order.
Sources: doc/relp.h:23-35, src/sendbuf.h:43-55
Outbound Data Structures
The buffering strategy utilizes several key data structures to manage outbound traffic. The primary unit of buffering is the relpSendbuf_t, which holds a single RELP frame ready for transmission.
relpSendbuf_t
The relpSendbuf_t structure tracks the state of an individual frame. It includes pointers to the raw data, the transaction number (txnr), and a callback for handling the server's response.
Sources: src/sendbuf.h:43-55
| Field | Type | Description |
|---|---|---|
pData | relpOctet_t* | The raw frame data as it appears on the wire. |
txnr | relpTxnr_t | The unique transaction number for this frame. |
rspHdlr | function pointer | Callback triggered when the response for this txnr arrives. |
bufPtr | size_t | Tracks how many octets have been successfully sent. |
lenData | size_t | Total length of the buffered data. |
relpSendq_t
For server-side operations or complex client scenarios, librelp uses a relpSendq_t (Send Queue). This is a thread-safe doubly-linked list of relpSendqe_t entries, each wrapping a relpSendbuf_t.
Sources: src/sendq.c:98-111, src/relp.h:92-105
The diagram shows the relationship between the queue, its entries, and the underlying data buffers.
Sources: src/sendq.c:45-55, src/relp.h:92-105
Transmission Logic
The buffering strategy distinguishes between "sending" (placing data on the wire) and "acknowledging" (removing data after a successful response).
Non-blocking Queue Transmission
The function relpSendqSend iterates through the send queue, attempting to write as much data as possible to the TCP socket. If a write is partial or would block (RELP_RET_EAGAIN), the bufPtr in the current relpSendbuf_t is updated, and the operation yields to the event loop.
Sources: src/sendq.c:237-268, src/sendbuf.c:103-125
Blocking Client Transmission
Clients typically use relpSendbufSendAll, which is a blocking call that ensures the entire buffer is written to the OS network stack. It utilizes a timeout mechanism based on the session's configured timeout.
Sources: src/sendbuf.c:139-183
Reliability and Unacked Buffers
A critical component of the strategy is the Unacked List. After a client successfully sends a buffer, it is not immediately discarded. Instead, it is moved to the session's pUnackedLstRoot.
Sources: src/relpsess.h:116-118, src/sendbuf.c:191-196
The flow shows the lifecycle of a data buffer from creation through its status as an "unacked" entry until final acknowledgement.
Sources: src/sendbuf.c:139-200, doc/relp.html: "Message Transmission"
Session Re-establishment
If a session is broken, the unacked list allows the client to resend all commands that were not acknowledged by the server. During re-establishment, buffers are re-sent from the unacked list but are not added back to it (as they are already present), preventing duplicates or queue corruption.
Sources: src/sendbuf.c:142-146, doc/relp.html: "High Latency Environment"
Configuration Parameters
The buffering behavior is influenced by several engine and session-level settings.
Sources: src/relp.h:182-198
| Parameter | Default | Description |
|---|---|---|
RELP_DFLT_WINDOW_SIZE | 128 | The number of unacked frames allowed before the window stalls. |
RELP_DFLT_MAX_DATA_SIZE | 128 KB | Maximum size of the data portion of a single frame. |
RELP_RCV_BUF_SIZE | 32 KB | Size of the buffer used for receiving data from the transport. |
Summary of Logic
The Data Buffering Strategy ensures that librelp can handle high-latency environments and sudden disconnections without losing data. By wrapping frames in relpSendbuf_t and maintaining them in an unacked list until the application-level "rsp" command is received, librelp provides a robust delivery guarantee. Servers benefit from the relpSendq structure, which allows them to continue processing and queuing responses even if the TCP window of a hibernated or slow client is full.
Sources: doc/relp.html: "Server Session Closure with a Hibernated Client", src/sendq.c:237-268