Encoders (H.264, MJPEG, JPEG)

The encoder system in picamera2 is responsible for compressing raw image frames from the camera into standard formats such as H.264 for video, and MJPEG or JPEG for video and still images. The architecture is designed to be flexible, supporting both hardware-accelerated encoders (via V4L2) and software-based encoders that leverage libraries like libav (PyAV) and simplejpeg. This allows for efficient encoding across different Raspberry Pi models and use cases.

All encoders inherit from a common Encoder base class, which defines a standard interface and lifecycle. This includes methods for starting, stopping, and processing frames. Encoders work in conjunction with Output objects, which handle the destination of the encoded data, whether it's a file, a network stream, or an in-memory buffer. The system also provides an optional, integrated audio encoding pipeline that runs in a separate thread, muxing audio and video into a final container format when using a compatible output like PyavOutput.

Core Encoder Architecture

The foundation of the encoding system is the Encoder base class, which provides a common API and manages the encoder's state. Different encoder implementations inherit from this class to provide specific compression functionality.

Sources: picamera2/encoders/encoder.py

Encoder Class Hierarchy

The system features several encoder types, each tailored for a specific format or backend. They form an inheritance structure as shown below.

This diagram shows the inheritance relationships between the main encoder classes.
Sources: picamera2/encoders/encoder.py, picamera2/encoders/h264_encoder.py, picamera2/encoders/mjpeg_encoder.py, picamera2/encoders/jpeg_encoder.py, picamera2/encoders/libav_h264_encoder.py, picamera2/encoders/libav_mjpeg_encoder.py

Encoder Lifecycle

The Encoder base class defines a clear lifecycle managed through start() and stop() methods.

1. Initialization: An encoder is instantiated. Properties like bitrate, qp, or profile can be set.
2. Start: The start(quality) method is called.
   • It first calls the internal _setup(quality) method, which translates a high-level Quality setting into specific parameters like bitrate.
   • It then calls the internal _start() method, where the specific encoder implementation performs its setup (e.g., configuring V4L2 controls or initializing libav codecs).
   • The encoder's output objects are also started.
   • The running flag is set to True.
3. Encoding: For each frame, the encode(stream, request) method is called. This method handles frame skipping and calls the internal _encode(stream, request) for the actual encoding logic.
4. Stop: The stop() method is called.
   • The running flag is set to False.
   • It calls the internal _stop() method to flush any remaining frames and release resources.
   • The encoder's output objects are stopped.

The following diagram illustrates the sequence when an encoder is started automatically by Picamera2.

Quality Configuration

To simplify configuration, the picamera2 library provides a Quality enum. When start_encoder or encoder.start is called with a quality argument, the encoder's _setup method maps this abstract level to concrete bitrate or quantisation parameter (QP) values. This provides a consistent way to control output quality across different encoders.

Sources: picamera2/encoders/encoder.py:12-23

The specific mappings are defined within each encoder class. For example:

Sources: picamera2/encoders/h264_encoder.py:97-108, picamera2/encoders/libav_h264_encoder.py:60-71, picamera2/encoders/mjpeg_encoder.py:24-35, picamera2/encoders/jpeg_encoder.py:55-66, picamera2/encoders/libav_mjpeg_encoder.py:27-38

Encoder Implementations

Hardware-Accelerated Encoders (V4L2)

These encoders use the Video4Linux2 (V4L2) API to access the Raspberry Pi's hardware video codecs, offering the best performance with low CPU usage. They inherit from V4L2Encoder.

H264Encoder

The H264Encoder is the primary hardware-accelerated H.264 encoder. It provides fine-grained control over the encoding process.

• Key Parameters:
  • bitrate (int): Target bitrate for the encoded stream.
  • iperiod (int): The interval between I-frames (keyframes).
  • qp (int): Sets a fixed Quantisation Parameter for constant quality mode, disabling constant bitrate.
  • profile (str): Sets the H.264 profile (e.g., "baseline", "main", "high").
  • repeat (bool): If true, repeats SPS/PPS headers for every I-frame, which is useful for streaming.
• Functionality: It configures the hardware by setting V4L2 controls such as V4L2_CID_MPEG_VIDEO_H264_PROFILE and V4L2_CID_MPEG_VIDEO_H264_I_PERIOD. It also automatically adjusts the H.264 level based on resolution, framerate, and bitrate to ensure compliance.

Sources: [picamera2/encoders/h264_encoder.py]

MJPEGEncoder

The MJPEGEncoder uses the hardware MJPEG codec. It is simpler than the H.264 encoder and is primarily configured via its bitrate. It will raise a RuntimeError if instantiated on a platform where the hardware MJPEG encoder is not available.

Sources: [picamera2/encoders/mjpeg_encoder.py]

Software-Based Encoders

Software encoders run on the CPU and are used on platforms without hardware acceleration (like the Raspberry Pi 5) or for formats not supported by hardware.

Libav Encoders (H.264 & MJPEG)

These encoders are wrappers around the PyAV library, which provides Python bindings for FFmpeg's libav. They are the default H.264 and MJPEG encoders on Pi 5.

• LibavH264Encoder: A software H.264 encoder. It can also be configured to use the V4L2 hardware encoder on VC4 platforms by setting use_hw = True. It supports parameters like bitrate, iperiod, qp, and profile.
  Sources: [picamera2/encoders/libav_h264_encoder.py]
• LibavMjpegEncoder: A software MJPEG encoder. It is configured using bitrate or qp.
  Sources: [picamera2/encoders/libav_mjpeg_encoder.py]

Both Libav encoders work by creating an av.VideoFrame from the camera buffer and passing it to the libav codec for encoding.

JpegEncoder

The JpegEncoder is a highly optimized, multi-threaded software encoder for creating JPEG images.

• Backend: It uses the simplejpeg library.
• Key Parameters:
  • num_threads (int): Number of threads to use for encoding.
  • q (int): JPEG quality level (1-100).
  • colour_subsampling (str): Can be '420', '422', or '444'.
• Features: A key feature is its ability to encode directly from the YUV420 camera stream format, which is significantly faster and uses less memory than converting to RGB first.

Sources: picamera2/encoders/jpeg_encoder.py:36-43

Usage Patterns

Encoders can be used in two primary ways: automatically managed by the Picamera2 object, or manually controlled by the user.

Sources: [tests/codecs.py]

Automatic Usage

This is the simplest and most common method. The Picamera2 object handles the encoder's lifecycle.

# From tests/codecs.py
picam2 = Picamera2()
config = picam2.create_video_configuration()
picam2.configure(config)
picam2.start()

# Start the encoder
picam2.start_encoder(H264Encoder(), output="test.h264", quality=Quality.HIGH)
time.sleep(5)
picam2.stop_encoder()

Sources: tests/codecs.py:30-34, [tests/quality_check.py:53-56]

Manual Usage

Manual control provides more flexibility, such as sending encoded frames to multiple outputs or performing actions between frames.

# From tests/codecs.py
h264_encoder = H264Encoder()
h264_encoder.size = config["main"]["size"]
h264_encoder.format = config["main"]["format"]
h264_encoder.bitrate = 5000000
h264_encoder.output = FileOutput("test.h264")

h264_encoder.start()
start_time = time.time()
while time.time() - start_time < 5:
    request = picam2.capture_request()
    h264_encoder.encode("main", request)
    request.release()
h264_encoder.stop()

Sources: tests/codecs.py:15-24, [tests/codecs.py:36-42]

The following diagram shows the manual encoding process.

Audio Integration

The base Encoder class includes functionality for encoding an audio stream alongside the video. This is disabled by default but can be enabled by setting encoder.audio = True.

• How it works: When enabled, a dedicated thread (_audio_thread_func) is started. This thread captures audio using pyav, encodes it, and passes the audio packets to the outputframe method of the associated Output object.
• Requirements: This feature requires an Output class that can handle multiplexing audio and video streams, such as PyavOutput.
• Configuration:
  • audio (bool): Set to True to enable.
  • audio_input (dict): PyAV parameters for opening the audio input device (e.g., {'file': 'default', 'format': 'pulse'}).
  • audio_output (dict): PyAV parameters for the output audio stream (e.g., {'codec_name': 'aac'}).
  • audio_sync (int): A microsecond value to adjust audio/video synchronization.

The diagram below illustrates the parallel processing of audio and video frames.

This diagram shows how video frames from the camera and audio from a separate device are encoded in parallel threads and sent to the same output for muxing.
Sources: picamera2/encoders/encoder.py:61-71, picamera2/encoders/encoder.py:228-240, picamera2/encoders/encoder.py:291-308