Audio

Audio tracks work like video tracks. Declare them on your Output, emit samples alongside frames, and the runtime handles encoding, transport, and synchronization.

Declaring an audio track

from reactor_runtime import Audio, Output, Video


class MyOutput(Output):
    main_video: Video
    main_audio: Audio

Emit both together from the run loop:

async def run(self) -> None:
    while True:
        await self.connected.wait()
        while self.connected.is_set():
            video = self.pipe.forward(prompt=self.prompt)
            audio = self.pipe.generate_audio()
            await self.emit(
                MyOutput(
                    main_video=video,  # (H, W, 3) uint8
                    main_audio=audio,  # (1, N) int16
                )
            )

Every track declared on the class must be supplied on every emit(). When a tick produces no audio, send an empty array rather than omitting the field:

silence = np.zeros((1, 0), dtype=np.int16)

Audio format

Property	Value
dtype	`int16`
Shape	`(channels, samples)` — `(1, N)` for mono
Sample rate	48,000 Hz
Channels	1 (mono)

The runtime encodes to Opus for transport. 48 kHz is the Opus native rate, so audio already at 48 kHz needs no resampling.

Custom sample rates

If your model generates at a different rate, subclass Audio and declare it:

from reactor_runtime import Audio


class Audio16k(Audio):
    sample_rate = 16_000


class MyOutput(Output):
    main_video: Video
    main_audio: Audio16k

The runtime resamples to 48 kHz for transport. Declaring the true rate matters — it is what makes the resampling correct and keeps playback at the right pitch and duration.

Keeping audio and video in sync

When you emit an output carrying both tracks, the runtime keeps them aligned. If the video payload is a batch, the audio is split proportionally across those frames. The rule that follows: each emit should carry the audio that belongs to its video. At 30 fps, one video frame is 48000 / 30 = 1600 samples, so a batch of 3 frames should carry about 4,800 samples (100 ms). Emitting a frame’s worth of video with a second’s worth of audio will drift, however well the transport behaves.

Rate-matching two streams

Models whose audio and video come from separate producers need to pair them before emitting. Buffer whichever stream runs ahead, and emit only when both have material for the same span:

async def run(self) -> None:
    while True:
        await self.connected.wait()
        audio_pending: dict[int, np.ndarray] = {}
        video_pending: dict[int, np.ndarray] = {}
        next_block = 0

        while self.connected.is_set():
            kind, index, data = await self._next_block()
            if kind == "audio":
                audio_pending[index] = data
            else:
                video_pending[index] = data

            # ✅ Emit only once both tracks have the same block
            while next_block in audio_pending and next_block in video_pending:
                await self.emit(
                    MyOutput(
                        main_video=video_pending.pop(next_block),
                        main_audio=audio_pending.pop(next_block),
                    )
                )
                next_block += 1

Pairing by block index rather than arrival order is what keeps the two tracks locked together when one producer is briefly slower than the other.

Reading audio from the client

Inbound audio arrives as (1, samples) int16 at 48 kHz, read the same way as video but in FIFO order so no samples are dropped:

from reactor_runtime import ReadMode

chunks = self.input.mic.try_read(1, mode=ReadMode.FIFO)

See Media Input for why audio wants FIFO and video wants the newest frame.

Media Input

Read the client’s camera and microphone.

Recording

Capture the model’s output into downloadable clips.

Get Started

Build a Model

Media & Transport

Releases

Declaring an audio track

Audio format

Custom sample rates

Keeping audio and video in sync

Rate-matching two streams

Reading audio from the client

Next

Media Input

Recording

​Declaring an audio track

​Audio format

​Custom sample rates

​Keeping audio and video in sync

​Rate-matching two streams

​Reading audio from the client

​Next

Media Input

Recording

Declaring an audio track

Audio format

Custom sample rates

Keeping audio and video in sync

Rate-matching two streams

Reading audio from the client

Next