Block I/O Layer

Relevant source files

The following files were used as context for generating this wiki page:

• block/blk-core.c
• block/blk-mq.c
• Documentation/block/blk-mq.rst
• Documentation/kernel-hacking/hacking.rst
• Documentation/process/4.Coding.rst

The Block I/O Layer manages the transfer of data between the system memory and block storage devices. It provides an abstraction that allows the kernel to interact with various storage hardwares using a unified interface. This layer handles request queuing, scheduling, and dispatching to hardware-specific drivers.

Operations on the block device layer always execute in user context. Sources: Documentation/kernel-hacking/hacking.rst:43-44

Multi-Queue Block I/O (blk-mq) Architecture

The blk-mq architecture addresses the scalability limitations of single-queue designs in modern high-performance storage devices like NVMe SSDs. It uses a two-level queuing hierarchy to reduce lock contention and improve IOPS performance on multi-core systems. Sources: Documentation/block/blk-mq.rst

Queuing Structure

The system distributes I/O load across two distinct levels:

• Software Staging Queues: These are per-CPU or per-node queues where requests are initially placed. This locality reduces cache misses and cross-CPU locking.
• Hardware Dispatch Queues: These map to the actual hardware submission queues of the storage controller.

This diagram illustrates the flow of I/O requests from individual CPUs through software staging queues into hardware dispatch queues. Sources: Documentation/block/blk-mq.rst

Core Data Structures

The block layer relies on specific structures to manage the lifecycle of an I/O request. Sources: block/blk-core.c, block/blk-mq.c

Structure	Description
struct request	Represents a single I/O operation (read/write) sent to the device.
struct request_queue	The central structure managing the state and settings of a block device's queue.
struct bio	The basic unit of block I/O, representing one or more pages of data for transfer.
struct blk_mq_hw_ctx	The hardware context representing a hardware dispatch queue.
struct blk_mq_ctx	The software context representing a per-CPU staging queue.

Sources: block/blk-core.c, block/blk-mq.c, Documentation/block/blk-mq.rst

Request Lifecycle and Dispatch

The block layer processes I/O through a series of stages, starting from the filesystem and ending at the device driver. Sources: block/blk-mq.c

1. Submission: The upper layers submit a struct bio.
2. Mapping: The block layer maps the bio to a struct request and assigns it to a software context (blk_mq_ctx) based on the current CPU.
3. Queueing: Requests are stored in the software staging queue or sent directly to hardware if possible.
4. Dispatch: The blk-mq scheduler or dispatch logic moves requests from software queues to the hardware context (blk_mq_hw_ctx).
5. Execution: The hardware driver processes the request from the hardware queue.
6. Completion: The driver signals completion via an interrupt, and the block layer performs cleanup.

The sequence shows the standard path of an I/O request from submission to completion. Sources: block/blk-mq.c, block/blk-core.c

Resource Management and Safety

Locking

To prevent data corruption in multiprocessor systems, any resource accessed concurrently by more than one thread must be protected by a lock. The block layer uses various primitives, including spinlocks and mutexes, to protect queue structures and hardware registers. Sources: Documentation/process/4.Coding.rst:196-198

Memory Allocation

Block I/O operations often require dynamic memory. The layer uses kmalloc with specific flags:

• GFP_KERNEL: Used in user context for reliable allocation; may sleep.
• GFP_ATOMIC: Used if the allocation must occur without sleeping, such as inside an interrupt-safe region.

Sources: Documentation/kernel-hacking/hacking.rst:228-239

Conclusion

The Block I/O Layer provides a high-performance, scalable framework for storage operations. By utilizing the blk-mq architecture, the kernel minimizes CPU contention and maximizes the throughput of modern storage devices. The use of per-CPU software queues ensures that I/O processing scales effectively with the number of processor cores. Sources: Documentation/block/blk-mq.rst, block/blk-mq.c