Drawing Overlays

Relevant source files

The following files were used as context for generating this wiki page:

• examples/overlay_drm.py
• examples/overlay_qt.py
• picamera2/previews/q_gl_picamera2.py
• picamera2/previews/drm_preview.py
• examples/tensorflow/segmentation.py
• examples/imx500/imx500_segmentation_demo.py
• picamera2/request.py

The picamera2 library provides a powerful feature for drawing semi-transparent graphics on top of the live camera preview. This is accomplished using the set_overlay method, which is supported by multiple preview backends, including Qt (OpenGL) and DRM (Direct Rendering Manager). This functionality is particularly useful for annotating the camera feed with dynamic information such as timestamps, sensor data, or the results of real-time AI inference like bounding boxes and segmentation masks.

To use overlays, the camera must be configured with a buffer count of at least 2. The overlay itself is a NumPy array representing an RGBA image, where the alpha channel controls the transparency of the overlay. This allows for flexible and efficient rendering of graphics directly on the preview window without modifying the underlying captured image data.

Core Concept: set_overlay

The primary method for drawing overlays is picam2.set_overlay(overlay). This method is available on the Picamera2 object but delegates the work to the active preview window implementation.

Overlay Parameter

The overlay argument passed to set_overlay must be a NumPy array with a specific format.

Property	Type	Description
overlay	np.ndarray or None	A NumPy array of shape (height, width, 4) and dtype=np.uint8. This represents an RGBA image. Passing None will remove the current overlay.

The four channels of the array correspond to Red, Green, Blue, and Alpha. The Alpha channel value (0-255) determines the transparency of each pixel.

Sources: picamera2/previews/q_gl_picamera2.py:279-281, picamera2/previews/drm_preview.py:114-116

A critical prerequisite for using overlays is that the camera configuration must have buffer_count set to 2 or more. This is because the preview system needs to hold onto buffers while the overlay is being applied.

Sources: picamera2/previews/q_gl_picamera2.py:283-284, picamera2/previews/drm_preview.py:112-113

Overlay Implementations

The picamera2 library supports overlays on different preview systems, primarily Qt-based previews and DRM-based previews. The underlying mechanism differs significantly between them.

Qt (OpenGL) Overlay: QGlPicamera2

When using a QGlPicamera2 preview window, overlays are handled using OpenGL textures.

The set_overlay method in QGlPicamera2 performs the following steps:

1. It acquires a lock to ensure thread safety.
2. It makes the EGL context current.
3. If the overlay is None, it sets self.overlay_present to False and repaints the window to remove the overlay.
4. If an overlay array is provided, it binds an OpenGL texture (self.overlay_texture) and loads the RGBA data from the NumPy array into it using glTexImage2D.
5. It sets self.overlay_present to True and repaints the window.

The repaint method is responsible for drawing. It first draws the main camera image texture, and then, if self.overlay_present is true, it draws the overlay texture on top of it. Blending is enabled (glEnable(GL_BLEND)) to handle the transparency from the alpha channel.

Sources: picamera2/previews/q_gl_picamera2.py:279-305, picamera2/previews/q_gl_picamera2.py:307-332

The following diagram illustrates the sequence of setting an overlay in a Qt application.

DRM Overlay: DrmPreview

The Direct Rendering Manager (DRM) implementation uses hardware planes for overlays, which can be very efficient.

The set_overlay method in DrmPreview works as follows:

1. It checks that overlays are supported by verifying the existence of an overlay_plane.
2. If the overlay is None, it clears the reference to the overlay framebuffer.
3. If an overlay array is provided, it creates a new pykms.DumbFramebuffer with the AB24 (RGBA) pixel format.
4. The RGBA data from the NumPy array is written into this framebuffer using a memory map (mmap).
5. The render_drm method is then triggered (or will be on the next frame) to display the overlay.

In render_drm, an atomic request (pykms.AtomicReq) is used to commit changes to the display. It adds a plane for the main camera image and, if an overlay framebuffer exists, it adds a second plane (self.overlay_plane) for the overlay. This hardware plane is rendered on top of the primary plane.

Sources: picamera2/previews/drm_preview.py:108-128, picamera2/previews/drm_preview.py:185-192

The following diagram shows the process of creating and displaying a DRM overlay.

Practical Examples

The picamera2 repository includes several examples demonstrating how to create and use overlays.

Example File	Use Case	Key Technique -
examples/overlay_qt.py	Displaying a timestamp on a Qt preview.	Uses cv2.putText to draw the current time on a transparent NumPy array, which is then passed to picam2.set_overlay(). -
examples/overlay_drm.py	Displaying a timestamp on a DRM preview.	Similar to the Qt example, but for a DRM/KMS preview window. It also uses cv2.putText to generate the overlay image. -
examples/tensorflow/segmentation.py	Visualizing image segmentation results.	After running inference, it creates a colored overlay from the segmentation mask. Each class index is mapped to a color. The combined masks are then displayed using picam2.set_overlay(). The alpha channel is adjusted for transparency. -
examples/imx500/imx500_segmentation_demo.py	Visualizing segmentation on IMX500.	This is a device-specific example for the IMX500 AI sensor. It creates masks from the model's output tensor, assigns colors, and combines them into a single overlay array which is passed to picam2.set_overlay(). It demonstrates a real-world AI application.

Creating a Timestamp Overlay

A common use case is drawing text, like a timestamp, onto the preview. This is done by creating a transparent RGBA image and drawing the text onto it.

# From examples/overlay_qt.py
# Create a transparent overlay
overlay = np.zeros((300, 400, 4), dtype=np.uint8)
# Draw the timestamp on the overlay
text = time.strftime("%Y-%m-%d %X")
cv2.putText(overlay, text, (10, 30), font, 1, (255, 255, 255, 255), 2)
# Set the overlay
picam2.set_overlay(overlay)

Sources: examples/overlay_qt.py:27-32

Creating a Segmentation Mask Overlay

In AI applications, overlays are used to visualize model outputs. For segmentation, each pixel is classified, and an overlay can be created by coloring each class.

# From examples/tensorflow/segmentation.py
# ... after getting segmentation mask from model ...
overlay = np.zeros((normalSize[1], normalSize[0], 4), dtype=np.uint8)
global masks  # noqa
for v in masks.values():
    overlay += v
# Set Alphas and overlay
overlay[:, :, -1][overlay[:, :, -1] == 255] = 150
picam2.set_overlay(overlay)

Sources: examples/tensorflow/segmentation.py:108-115

In this example, masks is a dictionary where each value is an RGBA image for a specific detected class. These are summed to create a final composite overlay. The alpha channel is then adjusted to make the overlay semi-transparent.

Alternative: Drawing Directly on Buffers

An alternative to set_overlay is to directly modify the pixel data of the camera buffer. This can be done using a post_callback or by mapping the buffer of a CompletedRequest. This approach modifies the actual image data, which means the annotations will be present in saved files or encoded video streams. This is different from set_overlay, which only affects the live preview.

The MappedArray class is used to get a writable NumPy array that points to the camera buffer's memory.

# This is a different technique, NOT using set_overlay
# From examples/tensorflow/yolo_v5_real_time_with_labels.py
def DrawRectangles(request):
    with MappedArray(request, "main") as m:
        for rect in rectangles:
            # ... calculate rect_start, rect_end ...
            cv2.rectangle(m.array, rect_start, rect_end, (0, 255, 0, 0))
            # ... draw text ...

Sources: examples/tensorflow/yolo_v5_real_time_with_labels.py:46-57, picamera2/request.py:48-77

This method is useful when the annotations need to be permanently part of the output image.

Summary

Drawing overlays in picamera2 is a flexible feature for annotating live previews. The set_overlay method provides a high-level interface that works across both Qt (OpenGL) and DRM preview backends. It operates by rendering a user-provided RGBA NumPy array on top of the camera image, making it ideal for displaying transient information like timestamps or AI model results without altering the source image. For permanent annotations, direct buffer modification via MappedArray offers an alternative.