Using Hardware Accelerators (Hailo, IMX500)

Relevant source files

The following files were used as context for generating this wiki page:

• picamera2/devices/hailo/hailo.py
• picamera2/devices/imx500/imx500.py
• examples/hailo/detect.py
• examples/imx500/imx500_object_detection_demo.py
• examples/hailo/pose.py
• examples/imx500/imx500_segmentation_demo.py
• picamera2/request.py

The picamera2 library provides direct integration with hardware AI accelerators, such as the Hailo AI module and the Sony IMX500 sensor with its integrated AI processor. This allows for performing real-time machine learning inference directly on the image data as it flows through the camera pipeline, enabling high-performance, low-latency applications like object detection, pose estimation, and image segmentation.

This document outlines the architecture and usage patterns for both the Hailo and IMX500 accelerators within the picamera2 ecosystem. While both serve a similar purpose, their integration and programming models have distinct differences.

Hailo AI Accelerator

The Hailo integration uses a helper class that manages the AI model and inference process. The typical workflow involves capturing a low-resolution stream from the camera, running inference on it, and then overlaying the results on the high-resolution preview stream.

Sources: picamera2/devices/hailo/hailo.py, examples/hailo/detect.py, examples/hailo/pose.py

Architecture and Workflow

The Hailo class acts as a context manager to handle the model loading and resource management. The inference is performed on a low-resolution (lores) stream configured to match the input dimensions of the neural network model.

The general workflow is as follows:

1. Instantiate the Hailo class with the path to the model's HEF file.
2. Retrieve the model's required input shape using hailo.get_input_shape().
3. Configure picam2 with a main stream for preview and a lores stream matching the model's input size.
4. Start the camera and a preview window.
5. In a loop, capture the lores frame as a NumPy array.
6. Pass the array to hailo.run() to perform inference.
7. Post-process the inference results.
8. Use a pre_callback (draw_objects in the examples) to overlay visualizations (e.g., bounding boxes) onto the main stream before it is displayed.

This diagram illustrates the data flow for Hailo-based inference.

Key Components

Class / Function	Description	Source File
Hailo(model_path)	A context manager class to load a Hailo model from a .hef file and run inference.	picamera2/devices/hailo/hailo.py
hailo.get_input_shape()	Returns the required input shape (height, width, channels) for the loaded model.	picamera2/devices/hailo/hailo.py
hailo.run(frame)	Executes the inference on the provided NumPy array (frame).	picamera2/devices/hailo/hailo.py
picam2.pre_callback	A callback executed before the frame is displayed, used for drawing overlays.	examples/hailo/detect.py:67

Example: Object Detection

The object detection example demonstrates how to configure the camera and process inference results.

1. Configuration: A lores stream is created with a size matching the model's input.

   # examples/hailo/detect.py:61-63
   main = {'size': (video_w, video_h), 'format': 'XRGB8888'}
   lores = {'size': (model_w, model_h), 'format': 'RGB888'}
   config = picam2.create_preview_configuration(main, lores=lores, controls=controls)
   

2. Inference Loop: The lores array is captured and passed to the Hailo device.

   # examples/hailo/detect.py:71-77
   while True:
       frame = picam2.capture_array('lores')
       # Run inference on the preprocessed frame
       results = hailo.run(frame)
       # Extract detections from the inference results
       detections = extract_detections(results, video_w, video_h, class_names, args.score_thresh)
   

3. Drawing Results: A pre_callback is used to draw bounding boxes on the main preview frame.

   # examples/hailo/detect.py:38-47
   def draw_objects(request):
       current_detections = detections
       if current_detections:
           with MappedArray(request, "main") as m:
               for class_name, bbox, score in current_detections:
                   x0, y0, x1, y1 = bbox
                   label = f"{class_name} %{int(score * 100)}"
                   cv2.rectangle(m.array, (x0, y0), (x1, y1), (0, 255, 0, 0), 2)
                   cv2.putText(m.array, label, (x0 + 5, y0 + 15),
                               cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 255, 0, 0), 1, cv2.LINE_AA)
   

   Sources: examples/hailo/detect.py

IMX500 Smart Camera

The Sony IMX500 integration is more deeply embedded, as the AI processor is part of the sensor itself. The IMX500 helper class is used to configure the device, load firmware, and retrieve inference results from the camera's metadata.

Sources: picamera2/devices/imx500/imx500.py, examples/imx500/imx500_object_detection_demo.py, examples/imx500/imx500_segmentation_demo.py

Architecture and Workflow

A key difference from Hailo is the initialization order: the IMX500 object must be instantiated before the Picamera2 object. The inference results are not returned from a function call but are embedded in the CompletedRequest metadata.

The general workflow is as follows:

1. Instantiate the IMX500 class with the path to the network's RPK file. This automatically finds the device.
2. Optionally, configure the NetworkIntrinsics to specify the task, inference rate, etc.
3. Instantiate Picamera2, passing imx500.camera_num to select the correct camera.
4. Configure and start picam2. A progress bar can be shown for firmware upload using imx500.show_network_fw_progress_bar().
5. Set a pre_callback or post_callback to handle each completed request.
6. Inside the callback, get metadata via request.get_metadata().
7. Retrieve the raw tensor data using imx500.get_outputs(metadata).
8. Post-process the tensor and draw results on the frame.

This diagram shows the IMX500 data flow.

Key Components

Class / Function	Description	Source File
IMX500(network_file)	Initializes the IMX500 device and loads the network firmware (.rpk). Must be called before Picamera2().	picamera2/devices/imx500/imx500.py:365
imx500.camera_num	Property to get the camera number for Picamera2 instantiation.	picamera2/devices/imx500/imx500.py:408
imx500.get_outputs(metadata)	Extracts and returns a list of NumPy arrays representing the model's output tensors from the request metadata.	picamera2/devices/imx500/imx500.py:601
imx500.network_intrinsics	An object of type NetworkIntrinsics that describes the network's properties and can be configured by the user.	picamera2/devices/imx500/imx500.py:416
imx500.set_inference_roi_abs()	Sets the region of interest for inference on the full sensor resolution.	picamera2/devices/imx500/imx500.py:651
imx500.convert_inference_coords()	Converts relative inference coordinates to the output image's coordinate space.	picamera2/devices/imx500/imx500.py:420
request.get_metadata()	Retrieves a dictionary of metadata from a CompletedRequest, which includes the CnnOutputTensor.	picamera2/request.py:151

NetworkIntrinsics Class

This class provides a structured way to define and validate the properties of the neural network. It uses JSON schema for validation.

This diagram shows the main properties of the NetworkIntrinsics class, which are used to control the AI model's behavior.
Sources: picamera2/devices/imx500/imx500.py:48-362

Example: Image Segmentation

The segmentation demo showcases how to retrieve and process the model's output to create and display masks.

1. Initialization: IMX500 is created first, followed by Picamera2.

   # examples/imx500/imx500_segmentation_demo.py:70-71
   # This must be called before instantiation of Picamera2
   imx500 = IMX500(args.model)
   
   # examples/imx500/imx500_segmentation_demo.py:87
   picam2 = Picamera2(imx500.camera_num)
   

2. Callback for Processing: A pre_callback is defined to handle each frame. It creates masks from the tensor output and then draws them.

   # examples/imx500/imx500_segmentation_demo.py:15-18
   def create_and_draw_masks(request: CompletedRequest):
       """Create masks from the output tensor and draw them on the main output image."""
       masks = create_masks(request)
       draw_masks(masks)
   

3. Extracting Tensors: Inside the callback, the output tensor is retrieved from the metadata and processed to generate segmentation masks.

   # examples/imx500/imx500_segmentation_demo.py:22-26
   def create_masks(request: CompletedRequest) -> Dict[int, np.ndarray]:
       """Create masks from the output tensor, scaled to the ISP output."""
       res = {}
       np_outputs = imx500.get_outputs(metadata=request.get_metadata())
       input_w, input_h = imx500.get_input_size()
   

4. Drawing Overlays: The generated masks are combined into a single overlay and displayed.

   # examples/imx500/imx500_segmentation_demo.py:47-49
   # No need to resize the overlay, it will be stretched to the output window.
   res[i] = overlay
   
   # examples/imx500/imx500_segmentation_demo.py:59
   picam2.set_overlay(overlay)
   

   Sources: examples/imx500/imx500_segmentation_demo.py

Summary of Differences: Hailo vs. IMX500

Feature	Hailo	IMX500
Initialization	Hailo object can be created anytime, typically as a context manager.	IMX500 object must be created before the Picamera2 object.
Inference Trigger	Explicitly call hailo.run(frame) on a captured NumPy array.	Inference is performed automatically by the sensor hardware.
Data Source	A lores stream is captured and fed to the run method.	Inference is performed internally; results are attached to metadata.
Result Retrieval	The run method returns the inference results directly.	Call imx500.get_outputs(metadata) on the request's metadata.
Model File	.hef (Hailo Executable Format)	.rpk (firmware package)
Device Selection	Automatic.	The camera_num from the IMX500 object must be passed to Picamera2.

Sources: picamera2/devices/hailo/hailo.py, picamera2/devices/imx500/imx500.py, examples/hailo/detect.py, examples/imx500/imx500_object_detection_demo.py

Conclusion

The picamera2 library provides powerful and flexible APIs for leveraging hardware AI accelerators. The Hailo integration offers a straightforward approach where inference is explicitly run on a camera stream, making it easy to understand and debug. The IMX500 integration is more deeply embedded, offering a potentially more efficient pipeline by processing data directly on-sensor and delivering results via metadata, which simplifies the application logic by abstracting away the explicit inference step. Both integrations enable developers to build sophisticated, real-time computer vision applications with minimal overhead.