Audio Track Separation
Relevant source files
The following files were used as context for generating this wiki page:
Audio Track Separation enables the concurrent capture of multiple distinct audio sources—such as system output and microphone input—into independent tracks within a single media container. This system ensures that different audio streams do not mix during recording, allowing for granular control during playback or post-production.
The architecture relies on AVAssetWriter to manage multiple AVAssetWriterInput instances, each mapped to a specific capture source. The system currently supports separating "system" audio (captured via ScreenCaptureKit) and "mic" audio (captured via AVCaptureSession).
Architecture and Components
The system separates audio by defining unique labels for each source and routing their respective CMSampleBuffer data to corresponding writer inputs.
Core Components
- MovieWriter: Manages the
AVAssetWriterlifecycle. It maintains a dictionary ofaudioInputskeyed by string labels. - SckCapture: Interfaces with
ScreenCaptureKitto retrieve system-level audio buffers. - MicCapture: Interfaces with
AVCaptureDeviceto retrieve microphone buffers. - runRecordingSession: The orchestration function that initializes capture sources and links them to the writer.
Sources: Sources/Spike/MovieWriter.swift:15-20, Sources/Spike/RecSystemCmd.swift:14-45
Data Flow Diagram
The following diagram illustrates how disparate audio sources are routed through the MovieWriter into separate tracks in the final output file.
The MovieWriter uses the provided labels to identify which internal input should receive a specific buffer.
Sources: Sources/Spike/MovieWriter.swift:96-121, Sources/Spike/RecSystemCmd.swift:42-53
Multi-Track Configuration
The system configures audio tracks using standard AAC settings. Each track is initialized with identical properties but remains logically separate in the file structure.
| Property | Value / Configuration |
|---|---|
| Codec | kAudioFormatMPEG4AAC |
| Channel Count | 2 (Stereo) |
| Sample Rate | 48,000 Hz |
| Quality | AVAudioQuality.high |
| Real-time | expectsMediaDataInRealTime = true |
Sources: Sources/Spike/MovieWriter.swift:47-53
Track Initialization Logic
When the MovieWriter is initialized, it iterates through an array of audioLabels. For each label, it creates a new AVAssetWriterInput and adds it to the writer.
// Sources/Spike/MovieWriter.swift:46-54
var inputs: [String: AVAssetWriterInput] = [:]
for label in audioLabels {
let a = AVAssetWriterInput(mediaType: .audio, outputSettings: [
AVFormatIDKey: kAudioFormatMPEG4AAC,
AVNumberOfChannelsKey: 2,
AVSampleRateKey: 48000,
AVEncoderAudioQualityKey: AVAudioQuality.high.rawValue,
])
a.expectsMediaDataInRealTime = true
writer.add(a)
inputs[label] = a
}
Synchronization and Anchoring
To ensure tracks are synchronized, the system uses a Presentation Time Stamp (PTS) anchoring mechanism. The recording session start time is determined by the first arriving sample (either video or audio).
- Session Start: The writer calls
startSession(atSourceTime:)using the PTS of the very first sample. - Sample Filtering: Any audio samples with a PTS occurring before the session start time are dropped to prevent synchronization drift.
- PTS Reporting: The system calculates the offset between the first video PTS and the first audio PTS for each track to verify alignment.
Sources: Sources/Spike/MovieWriter.swift:102-111, Sources/Spike/MovieWriter.swift:124-129
Synchronization Sequence
Validation and Inspection
The InspectCmd and Util modules provide tools to verify that audio separation was successful. These tools analyze the resulting file to ensure multiple tracks exist and contain non-silent data.
- Track Enumeration: The
inspectcommand loads all audio tracks from the asset and reports their count. - Level Analysis: The system calculates the Root Mean Square (RMS) and peak levels for each track independently.
- Track Mapping: Tracks are identified by index (e.g.,
audio[0],audio[1]) in the inspection output.
Sources: Sources/Spike/InspectCmd.swift:42-50, Sources/Spike/Util.swift:103-138
Summary Table: Track Metrics
The MovieWriter tracks the following metrics for each separate audio label:
| Metric | Description |
|---|---|
audioAppended | Total number of buffers successfully written to the track. |
audioDropped | Total number of buffers discarded due to timing or readiness issues. |
firstAudioPTS | The timestamp of the first sample written to the specific track. |
lastAudioPTS | The timestamp of the final sample written to the specific track. |
Sources: Sources/Spike/MovieWriter.swift:156-168
Conclusion
Audio Track Separation is implemented by mapping distinct capture sources (SckCapture, MicCapture) to labeled AVAssetWriterInput instances. By anchoring the recording session to a common PTS and maintaining separate metadata for each track, the system provides high-fidelity, multi-source recordings suitable for advanced audio processing. This architecture is verified through automated inspection tools that validate track presence and audio signal integrity.
Sources: Sources/Spike/RecSystemCmd.swift:87-105, Sources/Spike/InspectCmd.swift:12-25