Diffusion & Image Generation
Relevant source files
The following files were used as context for generating this wiki page:
Diffusion & Image Generation refers to the system within the RunAnywhere SDKs that provides image generation capabilities using diffusion models like Stable Diffusion. The system supports three primary modes: text-to-image, image-to-image, and inpainting. It operates through the C++ commons core, which dispatches requests to specialized engine plugins.
The primary engine for diffusion on Apple platforms is NeuRT, which utilizes the Core ML framework to serve diffusion_ops. On other platforms like Electron, the diffusion namespace is currently a facade that reports a missing backend until rac_diffusion_generate_proto is bound and a compatible backend is linked.
Sources: core/README.md:58, core/README.md:124, bindings/swift/Sources/RunAnywhere/CRACommons/include/rac_diffusion_types.h:46-50, bindings/electron/src/api/assets.ts:1149-1153
Architecture and Data Flow
The Diffusion system follows the standard RunAnywhere plugin architecture. The platform SDKs interact with the rac_diffusion_service (via the C ABI), which communicates with the plugin registry to find a suitable backend for the RAC_PRIMITIVE_DIFFUSION primitive.
The diagram shows the flow from a user request through the SDK and Core Commons to the hardware-accelerated NeuRT engine, then returning the resulting image data.
Sources: core/README.md:21-47, idl/diffusion_options.proto:114-124
Configuration and Initialization
You initialize the diffusion component using rac_diffusion_config_t. This configuration defines the model variant, memory limits, and tokenizer settings. Because compiled Core ML models on Apple platforms often lack tokenizer files like vocab.json and merges.txt, the SDK can automatically download them from specified sources.
Diffusion Configuration Parameters
| Parameter | Type | Description |
|---|---|---|
model_id | const char* | Optional identifier for the specific model. |
model_variant | rac_diffusion_model_variant_t | Specifies SD 1.5, 2.1, SDXL, etc. |
enable_safety_checker | rac_bool_t | Enables NSFW content filtering. |
max_memory_mb | int32_t | Sets maximum working-set memory in MiB. |
tokenizer | rac_diffusion_tokenizer_config_t | Configuration for vocabulary and merge file sourcing. |
Sources: bindings/swift/Sources/RunAnywhere/CRACommons/include/rac_diffusion_types.h:112-140
Supported Model Variants
- SD 1.5: Stable Diffusion 1.5, defaults to 512x512 resolution.
- SD 2.1: Stable Diffusion 2.1, defaults to 768x768 resolution.
- SDXL: Stable Diffusion XL, defaults to 1024x1024; requires 8GB+ RAM.
- SDXL Turbo: Optimized for fast generation with fewer steps.
- LCM: Latent Consistency Model, typically requiring only 4 steps.
Sources: bindings/swift/Sources/RunAnywhere/CRACommons/include/rac_diffusion_types.h:37-44
Generation Options and Modes
The generation mode is inferred based on the provided inputs. Providing only a text prompt triggers Text-to-Image mode. Adding a source image promotes the request to Image-to-Image. Including both a source image and a mask image triggers Inpainting mode.
This diagram illustrates how the system determines the diffusion mode based on input availability.
Sources: idl/diffusion_options.proto:62-65, bindings/swift/Sources/RunAnywhere/CRACommons/include/rac_diffusion_types.h:46-50
Generation Option Fields
| Field | Proto Type | Default | Description |
|---|---|---|---|
prompt | string | Required | Description of the desired image. |
negative_prompt | string | N/A | Things to avoid in the image. |
width / height | int32 | 0 (Default) | Resolution in pixels. |
steps | int32 | 0 (Default) | Denoising iterations (max 50). |
guidance_scale | float | 0.0 (Default) | Classifier-free guidance strength (max 20.0). |
seed | int64 | Random | Seed for reproducibility. |
strength | float | 0.75 | Denoising strength for Image-to-Image mode. |
output_format | enum | PNG | Encoding format (PNG or RAW_RGBA). |
Sources: idl/diffusion_options.proto:42-108, bindings/swift/Sources/RunAnywhere/CRACommons/include/rac_diffusion_types.h:155-212
Streaming and Progress Reporting
Generation progress is reported via the DiffusionStreamEvent proto or rac_diffusion_progress_t C struct. The system can report intermediate images during the denoising process if requested via options.
Streaming Sequence
This sequence diagram shows the asynchronous flow of progress events and the final image delivery.
Sources: idl/diffusion_options.proto:145-156, bindings/swift/Sources/RunAnywhere/CRACommons/include/rac_diffusion_types.h:217-238
Results and Safety
The DiffusionResult contains a list of generated images. Each DiffusionImage includes the raw bytes, resolution, and the specific seed used. A safety_flag indicates if the image was flagged by the safety checker for NSFW content, though this is advisory and not treated as a hard error.
The rac_diffusion_result_t structure in the C ABI provides metadata such as generation_time_ms for performance monitoring.
Sources: idl/diffusion_options.proto:124-143, bindings/swift/Sources/RunAnywhere/CRACommons/include/rac_diffusion_types.h:243-269
Summary
Diffusion & Image Generation in RunAnywhere provides a cross-platform interface for on-device image synthesis. By abstracting complexity into the NeuRT engine for Apple platforms and maintaining a consistent Protobuf-based IDL, it allows for high-quality, private image generation across supported devices. The system ensures efficiency by supporting optimized variants like SDXL Turbo and LCM while providing comprehensive control through denoising strength and guidance scale parameters.