Custom Encoders and Outputs

Relevant source files

The following files were used as context for generating this wiki page:

• picamera2/encoders/encoder.py
• picamera2/outputs/output.py
• picamera2/encoders/h264_encoder.py
• picamera2/encoders/libav_h264_encoder.py
• picamera2/encoders/jpeg_encoder.py
• picamera2/outputs/fileoutput.py
• picamera2/outputs/pyavoutput.py
• picamera2/request.py
• tests/codecs.py
• tests/quality_check.py
• CHANGELOG.md

The picamera2 library provides a flexible and extensible architecture for encoding camera frames and directing them to various outputs. This system is built upon two core abstract base classes: Encoder and Output. Users can leverage a variety of built-in implementations or create their own custom classes to handle specific encoding formats, containers, or streaming protocols. This modularity allows for powerful combinations, such as encoding a single camera stream into multiple formats simultaneously or directing output to files, network sockets, and complex multimedia containers.

The core design allows for the separation of concerns between the encoding process (compressing raw image data) and the output process (writing the compressed data to a destination). This makes it easy to swap out different encoders or outputs without altering the main application logic. For instance, the same application code can record to an H.264 file, an MJPEG stream, or a series of JPEG images simply by instantiating a different encoder class.

Core Concepts: Encoder and Output Classes

The foundation of the encoding system is the relationship between the Encoder and Output base classes. An Encoder takes raw video frames and compresses them, while an Output takes the compressed data from the encoder and writes it to a destination.

The Encoder Base Class

The Encoder class, defined in picamera2/encoders/encoder.py, provides the fundamental API that all specific encoder implementations must follow. It manages the encoder's state, properties, and the connection to one or more Output objects.

Key Properties:

Property	Description
width, height, size	The dimensions of the frames to be encoded.
stride	The stride (bytes per line) of the input frames.
format	The pixel format of the input frames (e.g., 'YUV420', 'RGB888').
output	A single Output object or a list of Output objects.
running	A boolean flag indicating if the encoder is currently active.

Core Methods:

• start(quality=None): Prepares the encoder and associated outputs. It calls the internal _setup(quality) and _start() methods for specific encoder logic.
• stop(): Stops the encoder and its outputs, flushing any remaining data. It calls the internal _stop() method.
• encode(stream, request): The main method called for each frame. It retrieves the frame buffer from the CompletedRequest and passes it to the internal _encode() method for processing.

The following diagram illustrates the class structure of the base Encoder.

The Output Base Class

The Output class, defined in picamera2/outputs/output.py, is a simple interface for handling data produced by an encoder.

Key Methods:

• start(): Sets the recording flag to True.
• stop(): Sets the recording flag to False.
• outputframe(frame, keyframe, timestamp, packet, audio): Abstract method to be implemented by subclasses. This is where the encoded data is received and processed.

Linking Encoders and Outputs

An Encoder is linked to its destination(s) via its output property. This property can be assigned a single Output instance or a list of them, enabling a single encoder to feed multiple destinations simultaneously.

Sources: picamera2/encoders/encoder.py:161-175, tests/split_output_test.py:18-19

The data flow from an encoder to its outputs is straightforward. When the encoder produces a compressed frame, it calls outputframe on each Output object in its _output list.

Encoder Lifecycle and Data Flow

The lifecycle of an encoder is managed by start() and stop() calls, which are typically invoked by picam2.start_encoder() and picam2.stop_encoder().

The following sequence diagram illustrates the process when using the high-level start_encoder method.

Frame Data Access

Encoders access the pixel data of a captured frame through the CompletedRequest object. To avoid unnecessary data copies, they use the MappedArray or _MappedBuffer context managers. These tools map the camera's memory buffer directly into a NumPy array, allowing the encoder to read the pixel data efficiently.

# From picamera2/encoders/libav_h264_encoder.py
def _encode(self, stream, request):
    # ...
    with MappedArray(request, stream) as m:
        frame = av.VideoFrame.from_numpy_buffer(m.array, format=self._av_input_format, width=self.width)
        frame.pts = timestamp_us
        for packet in self._stream.encode(frame):
            # ...
            self.outputframe(bytes(packet), packet.is_keyframe, timestamp=packet.pts, packet=packet)
    # ...

Sources: picamera2/encoders/libav_h264_encoder.py:168-175, picamera2/request.py:46-73

Built-in Encoder Implementations

picamera2 includes several encoder implementations for common formats.

Encoder Class	Underlying Library	Key Use Case	Source File
H264Encoder	V4L2 Kernel Interface	Hardware-accelerated H.264 encoding on older Raspberry Pi OS.	picamera2/encoders/h264_encoder.py
JpegEncoder	simplejpeg	Software-based JPEG encoding for stills or MJPEG video.	picamera2/encoders/jpeg_encoder.py
LibavH264Encoder	PyAV (libav/FFmpeg)	Software H.264 encoding, default on Pi 5. Supports hardware acceleration on VC4 platforms.	picamera2/encoders/libav_h264_encoder.py
LibavMjpegEncoder	PyAV (libav/FFmpeg)	Software MJPEG encoding, default on Pi 5.	picamera2/encoders/libav_mjpeg_encoder.py

Sources: CHANGELOG.md:162, picamera2/encoders/h264_encoder.py, picamera2/encoders/jpeg_encoder.py, picamera2/encoders/libav_h264_encoder.py

Quality Configuration

Most encoders accept a quality parameter during start() or start_encoder(). This is an enum of type Quality with values from VERY_LOW to VERY_HIGH. Inside the encoder's _setup method, this abstract quality level is translated into a concrete configuration, such as a bitrate for video encoders or a q (quantization) value for JPEG.

# From picamera2/encoders/encoder.py
class Quality(Enum):
    VERY_LOW = 0
    LOW = 1
    MEDIUM = 2
    HIGH = 3
    VERY_HIGH = 4

Sources: picamera2/encoders/encoder.py:10-19

For example, the H264Encoder maps these quality levels to different bitrates, adjusted for the resolution and framerate.

# From picamera2/encoders/h264_encoder.py
def _setup(self, quality):
    # ...
    quality = Quality.MEDIUM if quality is None else quality
    # These are suggested bitrates for 1080p30 in Mbps
    BITRATE_TABLE = {Quality.VERY_LOW: 2,
                     Quality.LOW: 4,
                     Quality.MEDIUM: 6,
                     Quality.HIGH: 9,
                     Quality.VERY_HIGH: 15}
    reference_complexity = 1920 * 1080 * 30
    actual_complexity = self.width * self.height * getattr(self, "framerate", 30)
    reference_bitrate = BITRATE_TABLE[quality] * 1000000
    self.bitrate = int(reference_bitrate * actual_complexity / reference_complexity)

Sources: picamera2/encoders/h264_encoder.py:104-118, tests/quality_check.py

Built-in Output Implementations

picamera2 provides several Output subclasses for common use cases.

Output Class	Description	Source File
FileOutput	Writes encoded data to a file, file-like object, or a network socket. Can split UDP datagrams.	picamera2/outputs/fileoutput.py
PyavOutput	Uses PyAV to mux video and audio streams into a multimedia container (e.g., MP4).	picamera2/outputs/pyavoutput.py
SplittableOutput	An output wrapper that allows seamlessly switching the destination file/output during a recording.	tests/split_output_test.py:18

FileOutput

This is the simplest output, writing raw encoded frames to a file or socket. It can be initialized with a file path or a file-like object. For UDP sockets, it can automatically split large frames into smaller datagrams.

Sources: picamera2/outputs/fileoutput.py:11-39, picamera2/outputs/fileoutput.py:100-111

PyavOutput

This output acts as a bridge to the PyAV library, enabling the creation of standard multimedia files like .mp4. It works by receiving encoded packets from an encoder and passing them to a PyAV container for muxing. It is essential for combining video and audio streams.

The PyavOutput must be informed of the streams it will receive from the encoder. This is done via the _add_stream method, which the encoder calls on its outputs.

Sources: picamera2/outputs/pyavoutput.py:19-33, picamera2/outputs/pyavoutput.py:41-50

Advanced Features

Audio Encoding

The base Encoder class has built-in support for capturing and encoding an audio stream alongside the video. This is handled in a separate thread.

Enabling Audio:

• Set encoder.audio = True.
• The Output must be able to handle audio, which currently means using PyavOutput.

Configuration:

• audio_input: A dictionary of parameters passed to av.open to select the audio source (e.g., {'file': 'default', 'format': 'pulse'}).
• audio_output: A dictionary of parameters to define the audio codec (e.g., {'codec_name': 'aac'}).
• audio_sync: A microsecond value to adjust audio/video synchronization.

The audio thread starts when the first video frame is encoded, captures audio packets, encodes them, and forwards them to the outputframe method of the attached Output(s) with the audio flag set to True.

Manual vs. Automatic Encoding

There are two primary ways to use an encoder.

1. Automatic (High-Level)
The easiest method is to use picam2.start_encoder(). This handles starting the encoder and feeding it frames from the camera's video stream automatically.

# From tests/codecs.py
def do_encode_auto(encoder, stream_name, output_name):
    print("Start encode by hand using", encoder, "to", output_name)
    picam2.start_encoder(encoder, output=output_name, name=stream_name)
    time.sleep(time_seconds)
    picam2.stop_encoder(encoder)
    print(encoder, "finished!")

Sources: tests/codecs.py:26-32

2. Manual (Low-Level)
For more control, you can manage the encoder "by hand". This involves manually starting the encoder, capturing requests in a loop, and calling encoder.encode() for each one.

# From tests/codecs.py
def do_encode_by_hand(encoder, stream_name, output_name):
    print("Start encode by hand using", encoder, "to", output_name)
    encoder.output = FileOutput(output_name)
    encoder.start()
    start = time.time()
    while time.time() - start < time_seconds:
        request = picam2.capture_request()
        encoder.encode(stream_name, request)
        request.release()
    encoder.stop()
    print(encoder, "finished!")

Sources: tests/codecs.py:13-24

Conclusion

The encoder and output system in picamera2 is a powerful, modular framework for processing video and image data. By separating the encoding logic from the output destination, it offers significant flexibility. Developers can choose from a range of built-in hardware and software encoders, direct output to files, sockets, or complex containers, and even extend the system with custom components to meet specialized requirements. Features like multi-output support and integrated audio encoding further enhance its capabilities, making it suitable for a wide array of camera applications.