Media and Audio

Play HTML5 video and audio in Views and route decoded audio to custom outputs.

You can play HTML5 <video> and <audio> elements inside Views in the Pro edition and higher. Video draws directly into the page, and decoded audio routes to an AudioOutput handler (a sound device or an engine mixer).

#The Default Audio Output

App::Create() installs an OS audio output on Windows, macOS, and Linux (on Linux, it uses PulseAudio when present and stays silent otherwise).

Renderer::Create() does not install an audio output. Providing one is optional— without one, video plays with no sound.

You can replace the default by setting a custom audio output on Platform before calling App::Create() or Renderer::Create()— see Setting Up the Platform.

#Playing Media

The library plays WebM media files (video/webm and audio/webm) encoded with VP9 video and Opus audio. Other containers and codecs aren't supported.

#Showing Playback Controls

To display built-in playback controls on a media element, add the controls attribute.

<video src="file:///ultralight-rocks.webm" controls></video>

The stylesheet and scripts for the built-in controls ship in the SDK's resources/ folder. The library loads them through the registered file system, so you only need to serve them— see Custom File System.

#Tuning Memory and Performance

You can tune the balance between memory footprint, playback smoothness, and seek responsiveness using Config::media_profile.

The library inspects this setting once when each media element begins loading. Changing Config::media_profile later affects only elements loaded after the change.

#Compiling Audio Output Code

The AudioOutput class and Platform::set_audio_output() are defined only when UL_HAS(MEDIA) is true. Free and Plus builds omit these APIs, so wrap audio output declarations in #if UL_HAS(MEDIA)— see Editions and Feature Macros.

You'll also need to include <Ultralight/platform/AudioOutput.h> directly. While <Ultralight/Ultralight.h> includes most platform interfaces, it leaves this header out.

#Implementing an Audio Output

To route decoded sound through an audio mixer or custom backend, subclass AudioOutput. The library assigns a unique stream_id to each loaded media file and uses that identifier in every subsequent call for the stream.

#Stream Lifecycle

Every stream moves through these calls in order:

#Handling Samples, Pausing, and Seeking

The library pushes audio as interleaved 32-bit float samples in the range [-1.0, 1.0] matching the sample rate and channel count given in CreateStream(). If the audio device requires a different rate or format, resample and downmix the samples in your handler.

When SetPaused() pauses a stream, keep any buffered samples so playback resumes from the same spot.

When Flush() runs, leave paused streams paused. The library resumes the stream once samples for the new position are ready.

#Example Implementation

The following example implements an AudioOutput handler for an engine mixer and installs it with Platform::instance().set_audio_output():

#include <Ultralight/Ultralight.h>
#include <Ultralight/platform/AudioOutput.h>

using namespace ultralight;

#if UL_HAS(MEDIA)

class MyAudioOutput : public AudioOutput {
 public:
  void CreateStream(uint32_t stream_id,
                    const AudioStreamFormat& format) override {
    ///
    /// Open a streaming source at format.sample_rate Hz with format.channels
    /// channels.
    ///
    // Pseudo-code, create a streaming source in your engine's mixer for
    // stream_id here.
  }

  void PushSamples(uint32_t stream_id, const float* samples,
                   uint32_t num_frames) override {
    ///
    /// Queue num_frames * channels interleaved floats. If the source's buffer
    /// is full, wait for room rather than dropping anything.
    ///
    // Pseudo-code, enqueue into your engine's streaming source here (blocking
    // when full).
  }

  void SetVolume(uint32_t stream_id, float volume) override {
    // Pseudo-code, set the source's gain to volume (0.0 to 1.0) here.
  }

  void SetPaused(uint32_t stream_id, bool paused) override {
    ///
    /// Pause keeps the buffered samples, so playback resumes where it left off.
    ///
    // Pseudo-code, pause or resume the source here.
  }

  void Flush(uint32_t stream_id) override {
    ///
    /// The element seeked. Drop everything queued and reset the consumed-frame
    /// counter to zero (leave the source paused if it was).
    ///
    // Pseudo-code, clear the source's queue and reset its frame counter here.
  }

  void Stop(uint32_t stream_id) override {
    ///
    /// End of stream. Let the buffered audio play out, and release any
    /// PushSamples() still waiting for room.
    ///
    // Pseudo-code, signal the source that no more samples are coming here.
  }

  void DestroyStream(uint32_t stream_id) override {
    // Pseudo-code, release the streaming source for stream_id here.
  }

  uint64_t GetPlaybackPosition(uint32_t stream_id) override {
    ///
    /// Frames the device has played since the last Flush(), straight from the
    /// hardware counter (return 0 if your backend can't tell).
    ///
    // Pseudo-code, return the source's consumed-frame counter here.
    return 0;
  }
};

MyAudioOutput my_audio_output;

void InitAudio() {
  ///
  /// Install the handler before creating the Renderer (it must outlive the
  /// Renderer).
  ///
  Platform::instance().set_audio_output(&my_audio_output);
}

#endif // UL_HAS(MEDIA)

#Threading and Backpressure

#Concurrency Across Threads

Calls to your AudioOutput implementation arrive from multiple threads, including the Renderer's thread and background decode threads. Never assume all calls for a stream occur on the same thread.

Because PushSamples() can run while Flush() or Stop() is called for the same stream, you must protect each stream's state. The only exception is DestroyStream()— the library calls Stop() first and waits for any running pushes to finish before destroying the stream.

🚧 Wait for Buffer Space

When a stream's buffer fills up, PushSamples() must wait for space rather than dropping samples. Waiting keeps audio decoding in step with device playback. If your output drops samples, the audio clock diverges from hardware playback and video drifts out of sync.

#Avoiding Deadlocks

A PushSamples() call waiting for room must return without writing its samples as soon as Flush() or Stop() is called for its stream. If the waiting call remains blocked, the library can hang.

You should never hold a stream mutex across that wait. Use an atomic flag that tells the waiting push to give up, and a separate lock for stream state— the implementations in the SDK's platform/ folder use this pattern.

#Syncing Video to Audio

The library calls GetPlaybackPosition() to keep video playback in sync with the audio track.

Return the number of frames the sound device has played on a stream since CreateStream() or the last Flush().

Overriding GetPlaybackPosition() is optional. The default implementation returns 0, which tells the library to time video with the system clock— playback sync is less precise without hardware timing.

#Counting Played Frames

Follow these rules when tracking the frame count:

• Count frames played by the audio device (adjusting for output latency if known), not frames pushed or queued in buffers.
• Reset the frame count to zero whenever Flush() runs.
• Stop incrementing the count while the stream is paused.