Accessing Image Data in Memory

Relevant source files

The following files were used as context for generating this wiki page:

• picamera2/request.py
• picamera2/picamera2.py
• examples/tensorflow/compositing.py
• examples/picamera2_multiprocessing.py
• examples/tensorflow/yolo_v5_real_time_with_labels.py
• apps/app_full.py

The picamera2 library provides powerful and flexible ways to access image data directly in memory, bypassing the need to save to a file. This is essential for applications involving real-time image processing, computer vision (with libraries like OpenCV or TensorFlow), or streaming. The primary mechanism for this is the CompletedRequest object, which encapsulates the image buffers and metadata for a completed camera capture.

This document covers the different methods for accessing this data, from high-level convenience functions that return copied arrays to low-level, zero-copy techniques for maximum performance. It also explores advanced usage, such as sharing memory buffers between different processes.

The CompletedRequest Object

The CompletedRequest is the central object for accessing captured frame data and metadata. It is returned by methods like capture_request() and is automatically released back to the camera system when no longer in use, for example, when its context manager exits.

Key CompletedRequest Methods

The CompletedRequest class provides several methods to access the image data in different formats.

Method	Return Type	Description
get_metadata()	Dict[str, Any]	Returns a dictionary of the metadata for the captured frame.
make_buffer(name)	np.ndarray	Returns a 1D NumPy array containing the raw buffer data for the specified stream.
make_array(name)	np.ndarray	Returns a multi-dimensional NumPy array representing the image. This is a copy of the buffer data.
make_image(name)	PIL.Image.Image	Returns a Pillow (PIL) Image object. This may or may not involve a data copy depending on the format.

Sources: picamera2/request.py:132-180

The following diagram illustrates the typical flow of capturing a request and accessing its data.

Sources: picamera2/picamera2.py:1303-1327, picamera2/request.py:138-180

Request Lifecycle and Management

A CompletedRequest has a reference count to manage its lifecycle. When you obtain a request (e.g., via capture_request()), you are responsible for releasing it using request.release() to prevent stalling the camera pipeline. The recommended way to handle this is with a context manager.

with picam2.captured_request() as request:
    # The request is valid within this block
    array = request.make_array("main")
    metadata = request.get_metadata()
# The request is automatically released here

Sources: picamera2/picamera2.py:1382-1387, picamera2/request.py:92-130

High-Level Capture Methods

For convenience, Picamera2 provides several high-level methods that wrap the capture_request() and data extraction logic. These methods are simpler to use but offer less control than working with a CompletedRequest directly.

• capture_array(name="main"): Captures a frame and returns it as a multi-dimensional NumPy array. This is equivalent to capture_request().make_array(name).
• capture_buffer(name="main"): Captures a frame and returns it as a 1D NumPy array (a flat buffer). This is equivalent to capture_request().make_buffer(name).
• capture_image(name="main"): Captures a frame and returns it as a PIL Image. This is equivalent to capture_request().make_image(name).

These methods are ideal for simple applications where you just need the image data in a standard format.

# Example using capture_array
image_array = picam2.capture_array("main")
print(image_array.shape)

# Example using capture_buffer
image_buffer = picam2.capture_buffer("main")
print(image_buffer.shape)

Sources: picamera2/picamera2.py:1515-1541, picamera2/picamera2.py:1404-1430, examples/tensorflow/yolo_v5_real_time_with_labels.py:101

Zero-Copy Access with MappedArray

For performance-critical applications like real-time video processing or machine learning inference, avoiding memory copies is crucial. The MappedArray context manager provides zero-copy access to the camera's underlying memory buffers.

When you enter the MappedArray context, you get a NumPy array that directly shares memory with the camera's DMA buffer. This array is only valid inside the with block. Once the block is exited, the buffer is unlocked, and the array data becomes invalid.

MappedArray Usage

You create a MappedArray from a CompletedRequest object.

with picam2.captured_request() as request:
    with MappedArray(request, "main") as m:
        # m.array is a NumPy array sharing memory with the camera buffer.
        # It can be read from or written to directly.
        cv2.rectangle(m.array, (x, y), (x + w, y + h), (0, 255, 0, 0))

Sources: picamera2/request.py:38-66, examples/tensorflow/compositing.py:37-46

The diagram below shows the relationship between CompletedRequest and MappedArray.

Sources: picamera2/request.py:23-66

Use Case: Real-time Inference

TensorFlow examples in the repository demonstrate using MappedArray (or capture_buffer which uses it internally) to efficiently feed image data to a model for inference, avoiding costly copies on every frame.

# From examples/tensorflow/real_time_with_labels.py
def DrawRectangles(request):
    with MappedArray(request, "main") as m:
        for rect in rectangles:
            # ... draw rectangle on m.array ...
            cv2.rectangle(m.array, rect_start, rect_end, (0, 255, 0, 0))

Sources: examples/tensorflow/real_time_with_labels.py:58-65

Advanced: Inter-Process Buffer Sharing

For complex applications, you might need to perform heavy processing in a separate process to avoid blocking the main camera loop. The picamera2 library supports this by sharing DMA buffer file descriptors (dmabuf fd) between processes. This is another form of zero-copy data sharing.

The examples/picamera2_multiprocessing.py file provides a Process class that demonstrates this pattern.

Workflow

The data sharing flow is as follows:

1. Main Process: Captures a CompletedRequest.
2. Main Process: Extracts the buffer's file descriptor (fd) and length from the request.
3. Main Process: Sends the fd, length, and any other arguments to the child process via a multiprocessing Queue.
4. Child Process: Receives the message. It uses a syscall (pidfd_getfd) to create a local file descriptor handle that points to the buffer in the main process's memory space.
5. Child Process: Uses mmap on the local fd to map the buffer into its own address space.
6. Child Process: Wraps the mapped memory in a NumPy array for processing.
7. Child Process: Optionally sends a result back to the main process, which then releases the original CompletedRequest.

The following sequence diagram illustrates this interaction.

Sources: examples/picamera2_multiprocessing.py

Implementation Details

Key functions from the example:

• Process.send(request, *args): In the main process, sends the buffer fd and length to the child.
• Process.capture_shared_array(): In the child process, waits for a buffer fd, maps it using mmap, and returns a NumPy array.
• Process._map_fd(picam2_fd): The internal function that uses pidfd_getfd to get a local handle to the parent's file descriptor.

Sources: examples/picamera2_multiprocessing.py:46-59, 83-111

Summary

picamera2 offers a tiered approach to accessing in-memory image data:

• High-Level Wrappers (capture_array, capture_image): Easiest to use, but involve data copies. Best for non-real-time applications or when simplicity is paramount.
• CompletedRequest Object: The standard method, offering a balance of control and ease of use. The context manager (with picam2.captured_request()) is the recommended pattern.
• MappedArray: Provides zero-copy access to camera buffers for maximum performance in single-process applications, especially for real-time analysis and drawing overlays.
• Inter-Process Sharing: The most advanced technique, enabling zero-copy data transfer to separate processes for parallel-heavy computation, preventing the main camera thread from blocking.