Batch Processing Queue
Relevant source files
The following files were used as context for generating this wiki page:
The Batch Processing Queue (formally known as the Olive Batch Serialization Queue) is a core module within Olive Studio designed to manage sequential optimization jobs. It allows users to execute parallel permutations or benchmark suites by queuing multiple Microsoft Olive recipes that target different models, execution providers, or optimization passes. Sources: src/components/features/BatchProcessingPanel.tsx:290, ABOUT.md:20
This system enables ML engineers and teams to automate complex optimization workflows, such as testing the same model across multiple hardware targets or comparing various quantization methods. It integrates directly with the backend execution engine via REST APIs and Server-Sent Events (SSE) to provide real-time status updates and performance metrics. Sources: README.md:104, ORIGINAL_REQUEST.md:46-52
Architecture and Data Flow
The Batch Processing Queue operates as a state-driven sequential engine. When a queue is started, the system iterates through all jobs marked with a queued status, converting their configurations into valid Olive JSON recipes before submission to the backend. Sources: src/components/features/BatchProcessingPanel.tsx:85-98
Job Execution Lifecycle
The following diagram illustrates the lifecycle of a job within the batch queue:
Explanation: The lifecycle of a batch job from creation through execution and metric collection. Sources: src/components/features/BatchProcessingPanel.tsx:85-188
Backend Integration
The system relies on three primary communication points with the Olive Studio backend:
- Job Submission: A
POSTrequest to/api/olive/runcontaining the generated recipe JSON. Sources: src/components/features/BatchProcessingPanel.tsx:111 - Log Streaming: A
GETrequest to/api/olive/stream/:jobIdwhich establishes a Server-Sent Events (SSE) connection. Sources: src/components/features/BatchProcessingPanel.tsx:135 - Status Monitoring: Jobs are tracked via unique IDs, allowing for real-time updates of logs and progress bars. Sources: ORIGINAL_REQUEST.md:33-35
Job Configuration and Creation
Jobs can be added to the queue through two primary interfaces: the Execution Workspace and the Batch Processing Panel. Sources: src/components/features/ExecutionWorkspace.tsx:610, src/components/features/BatchProcessingPanel.tsx:191
Interface Components
| Component | Functionality |
|---|---|
| Custom Job Form | Allows manual entry of Model Name, ID, Source Provider, and Target Execution Provider. Sources: src/components/features/BatchProcessingPanel.tsx:215-263 |
| Pass Pipeline Selector | Enables or disables specific Olive passes (Conversion, Quantization, Pruning, Attention Fusions) for the new job. Sources: src/components/features/BatchProcessingPanel.tsx:265-300 |
| Serial Engine Controls | Buttons to "Start Queue" (sequential processing) or "Halt Serial Engine" (closing active SSE connections). Sources: src/components/features/BatchProcessingPanel.tsx:325-343 |
| Queue List | A scrollable list of jobs showing real-time status (queued, running, completed, failed) and progress bars. Sources: src/components/features/BatchProcessingPanel.tsx:410-450 |
Data Structure: BatchJob
A batch job is represented by the BatchJob type, which contains the following critical fields:
id: Unique identifier (e.g.,job-1717540000000).status: Current state (queued,running,completed,failed).recipeJson: The serialized Olive recipe.progress: Numeric value representing completion percentage (parsed from stdout).logs: An array of strings containing raw output from the Olive CLI.metrics: Performance data extracted after completion (Latency, Throughput, VRAM, Compression).
Sources: src/components/features/BatchProcessingPanel.tsx:205-212, src/lib/recipePipeline.ts:47
Real-time Progress and Metrics
The queue utilizes active parsing of the Olive process output to provide feedback to the user.
Log Parsing Logic
The system monitors SSE "log" events for specific signals to update the UI:
- Progress Signals: Lines containing "Pass", "Step", or "%" are scanned for numeric values to update the job's progress percentage. Sources: src/components/features/BatchProcessingPanel.tsx:142-154
- Metric Signals: Upon successful completion (
exitCode: 0), the system executesparseOliveMetricsFromLogsto extract performance data. Sources: src/components/features/BatchProcessingPanel.tsx:178-183
Explanation: Sequence of events for real-time progress updates via Server-Sent Events. Sources: src/components/features/BatchProcessingPanel.tsx:111-185
Usage in Pipeline Validation
The Batch Processing Queue enforces strict validation rules before allowing a queue to start. If the current UI state or a specific job configuration violates compatibility rules (e.g., trying to run a GPU-only pass on a CPU provider), the "Start Queue" action is blocked. Sources: src/components/features/BatchProcessingPanel.tsx:90-94, src/lib/recipePipeline.ts:32-35
Validation Checks
- Schema Validation: Ensures the generated JSON follows the Microsoft Olive specification.
- Compatibility Validation: Checks if chosen passes are supported by the selected Execution Provider.
- Hardware Probe: Verifies that the local machine has the necessary drivers (CUDA, TensorRT, etc.) to execute the queued jobs.
Sources: src/lib/recipePipeline.ts:25-36, src/components/features/BatchProcessingPanel.tsx:64-71
Conclusion
The Batch Processing Queue is an essential automation layer in Olive Studio, transforming it from a simple recipe builder into a robust optimization workbench. By managing sequential jobs, streaming real-time logs, and extracting performance metrics, it provides a comprehensive environment for high-throughput model optimization across diverse hardware targets. Sources: ABOUT.md:20-23, README.md:110