OpenCV Integration

Relevant source files

The following files were used as context for generating this wiki page:

• examples/opencv_face_detect.py
• examples/opencv_mertens_merge.py
• examples/opencv_face_detect_2.py
• examples/opencv_face_detect_3.py
• apps/app_full.py
• examples/ocr.py
• examples/tensorflow/compositing.py
• examples/imx500/imx500_classification_demo.py

The picamera2 library provides seamless integration with OpenCV (cv2), a powerful open-source computer vision library. This allows developers to perform a wide range of image processing and computer vision tasks on the camera's video streams, from real-time object detection to advanced imaging techniques like High Dynamic Range (HDR) merging. The integration relies on OpenCV's ability to operate on NumPy arrays, which picamera2 can provide directly from the camera hardware.

The examples in the repository demonstrate that OpenCV is an optional but highly recommended dependency for advanced use cases. It is not installed as a core dependency of picamera2. The presence of OpenCV is checked in some applications to enable features like HDR.

Sources: apps/app_full.py:16-21

Core Concepts

Image Acquisition for OpenCV

The primary way to get image data from picamera2 into a format that OpenCV can use is by capturing frames as NumPy arrays.

• capture_array(): This method captures a frame from the specified stream and returns it as a NumPy array, which can be directly passed to OpenCV functions.
• capture_buffer(): For more efficient processing, especially with YUV formats, capture_buffer() can be used. The buffer can then be reshaped into a NumPy array. This is particularly useful for tasks like face detection on the luma (Y) channel of a YUV420 image.
• MappedArray: For annotating or modifying images in-place (e.g., drawing bounding boxes), picamera2 provides the MappedArray context manager. This maps the frame's buffer into a NumPy array that can be modified directly, with changes reflected in the preview or encoded output.

Sources: examples/opencv_mertens_merge.py:44, examples/opencv_face_detect_2.py:20, examples/ocr.py:21

Dual-Stream Processing

A common and efficient pattern is to configure picamera2 with two streams:

1. A high-resolution main stream for display or final capture.
2. A low-resolution lores stream for analysis.

This allows computationally intensive tasks (like face detection) to run on the smaller lores stream, improving performance. The results can then be scaled and drawn onto the main stream for visualization. The lores stream is often configured with a YUV420 format, as the grayscale 'Y' channel is sufficient for many detection algorithms and can be easily extracted.

Sources: examples/opencv_face_detect_2.py:23-25

Use Cases and Examples

The repository provides several examples showcasing different integrations with OpenCV.

Example Script	Purpose	Key OpenCV Functions Used
opencv_face_detect_2.py	Real-time face detection and drawing bounding boxes on the preview.	cv2.CascadeClassifier, detectMultiScale, cv2.rectangle
opencv_face_detect_3.py	Same as above, but draws boxes on a recorded H.264 video.	cv2.CascadeClassifier, detectMultiScale, cv2.rectangle
opencv_mertens_merge.py	Simple HDR imaging by merging three exposures.	cv2.createMergeMertens, merge.process
app_full.py	Advanced HDR imaging with multiple exposures and different merging algorithms.	cv2.createMergeDebevec, cv2.createMergeRobertson, cv2.createMergeMertens, cv2.createTonemap
ocr.py	Annotates the preview with text boxes found by an OCR library.	cv2.putText
tensorflow/compositing.py	Draws bounding boxes and creates image masks for compositing.	cv2.rectangle, cv2.putText, cv2.cvtColor, cv2.resize

Sources: examples/opencv_face_detect_2.py, examples/opencv_mertens_merge.py, apps/app_full.py, examples/ocr.py

Real-Time Face Detection

The opencv_face_detect_2.py example demonstrates an efficient pipeline for real-time face detection.

This diagram illustrates the data flow for real-time face detection.

1. Initialization: A cv2.CascadeClassifier is loaded with the Haar cascade model for frontal face detection.
2. Configuration: picam2 is configured with a main stream (e.g., 640x480) and a lores stream (e.g., 320x240, YUV420).
3. Callback: A post_callback function, draw_faces, is registered with picam2. This function will be called after a frame is captured but before it is displayed or encoded.
4. Capture Loop: The application enters a loop where it continuously calls picam2.capture_buffer("lores").
5. Processing: The captured YUV420 buffer is reshaped to extract the grayscale 'Y' plane. The face_detector.detectMultiScale method is run on this grayscale image to find faces.
6. Drawing: The draw_faces callback uses a MappedArray to get direct access to the main stream's image data. It iterates through the detected faces, scales their coordinates from the lores resolution to the main resolution, and uses cv2.rectangle to draw boxes around them.

This same principle is extended in opencv_face_detect_3.py to draw face boxes onto a video file being recorded with an H264Encoder.

Sources: examples/opencv_face_detect_2.py:15-41, examples/opencv_face_detect_3.py:16-46

High Dynamic Range (HDR) Imaging

OpenCV's photo module is used for creating HDR images by merging multiple frames taken at different exposure levels.

Simple Mertens Merge

The opencv_mertens_merge.py script shows a basic implementation:

1. Capture a frame to determine a "normal" exposure time.
2. Stop the camera and create a new configuration to allow manual control.
3. Capture three separate arrays with short, normal, and long exposure times by setting the ExposureTime and AnalogueGain controls.
4. Use cv2.createMergeMertens() to create a merge object.
5. Call merge.process() with the list of captured images.
6. The resulting floating-point image is clipped to the 0-255 range and saved.

Sources: examples/opencv_mertens_merge.py:15-56

Advanced HDR in app_full.py

The full application in apps/app_full.py implements a more sophisticated, non-blocking HDR capture and processing workflow.

The sequence diagram below shows the asynchronous process of capturing and processing HDR images.

This process involves:

1. Capturing a base image to determine the initial exposure.
2. Calculating a list of desired exposure times (e.g., -2, -1, 0, +1, +2 stops).
3. Disabling auto-exposure (AeEnable: False).
4. Sequentially capturing an image for each calculated exposure time by setting picam2.controls.ExposureTime.
5. Once all images are collected, a separate threading.Thread is dispatched to perform the merge operations.
6. The thread uses multiple OpenCV algorithms (Debevec, Robertson, Mertens) to create different HDR results and saves them to files.

Sources: apps/app_full.py:202-278, apps/app_full.py:281-338

Image Annotation

A very common use case for OpenCV is to draw information onto the camera preview. This is done by obtaining a writable NumPy array of the frame buffer and using OpenCV's drawing functions.

The draw_classification_results function in the IMX500 demo is a good example.

# From examples/imx500/imx500_classification_demo.py
def draw_classification_results(request: CompletedRequest, results: List[Classification], stream: str = "main"):
    """Draw the classification results for this request onto the ISP output."""
    with MappedArray(request, stream) as m:
        # ... coordinate calculations ...
        for index, result in enumerate(results):
            label = get_label(request, idx=result.idx)
            text = f"{label}: {result.score:.3f}"
            # ... more calculations ...
            cv2.putText(m.array, text, (text_x, text_y),
                        cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 1)

This pattern is repeated across many examples:

• A post_callback or pre_callback is defined.
• Inside the callback, with MappedArray(request, "main") as m: provides the NumPy array m.array.
• OpenCV functions like cv2.rectangle or cv2.putText are called to modify m.array in-place.

Sources: imx500/imx500_classification_demo.py:76-110, examples/ocr.py:21-27, examples/tensorflow/compositing.py:40-50