Multimedia and Streaming

Multimedia networking focuses on transmitting audio, video, and interactive data over packet-switched IP networks while coping with variable network delay, throughput fluctuations, and packet loss.

Multimedia Encoding

Digitisation - conversion of an analogue signal into digital samples.

Coding - representation of digitised media for storage or transmission.

Compression removes spatial and temporal redundancy. Codec design is covered in Multimedia Encodings. Audio Coding, Video Coding, and MPEG cover media-specific methods and standards.

Application Categories

Multimedia networking applications fall into three major categories based on interactive delay tolerance and playout constraints:

1. Streaming Stored Audio and Video:
   • Pre-recorded content hosted on servers (e.g., YouTube, Netflix, Hulu).
   • Client can begin playout shortly after request while remaining content continues downloading.
   • Supports user interactivity (pause, seek, fast-forward, rewind).
   • Tolerates initial buffering delays (2–10 seconds).
2. Conversational Voice and Video over IP:
   • Real-time human-to-human interaction (e.g., Skype, Zoom, Teams, VoIP).
   • Extremely strict delay requirements ($< 150\text{ms}$ preferred, $> 400\text{ms}$ unacceptable).
   • Cannot tolerate large buffering delays.
3. Streaming Live Audio and Video:
   • Live broadcast of events (e.g., sports events, twitch streams).
   • Playout cannot be fast-forwarded beyond live point.
   • Moderately strict delay tolerance (typically 3–10 seconds).

Streaming Stored Video

In stored video streaming, the client renders the video while the server streams the remaining portion over the network.

Continuous Playout Constraint

Once playback begins, the client must display frames continuously according to their original timing (e.g., 1 frame every $33\text{ ms}$ for 30 fps). If incoming data arrives slower than the playout rate, the client buffer empties, causing the video to freeze (buffer starvation).

Client-Side Buffering and Dynamics

To absorb network delay jitter and throughput variations, the client delays playout by $t_p$ seconds while filling an application buffer of capacity $B$.

Let:

• $r$ be the constant playout rate (bytes/sec).
• $x(t)$ be the variable incoming fill rate from the network at time $t$.
• $Q(t)$ be the buffer fill level at time $t$.

Packets from the network fill the buffer at rate $x(t)$. The client drains it at the constant playout rate $r$.

• Buffer Starvation ($\bar{x} < r$)
  If average network throughput drops below playout rate, $Q(t)$ decreases. If $Q(t) = 0$, playout freezes until the buffer refills to a threshold.
• Buffer Accumulation ($\bar{x} > r$)
  If average network throughput exceeds playout rate, the buffer fill level increases until saturated.

The choice of initial playout delay $t_p$ is a key trade-off:

• Larger $t_p$ reduces the probability of video freezing due to temporary network stalls.
• Smaller $t_p$ provides a faster startup response for the user.

Transport Mechanisms for Video Streaming

Streaming over UDP

• Server transmits packets at a constant rate matching the video encoding rate $r$.
• Encapsulated using RTP over UDP.
• Maintains short playout delays (2–5 seconds).
• Advantages
  Low overhead, simple implementation.
• Drawbacks
  Unfriendly to network congestion (does not back off); often blocked by corporate firewalls and NAT devices.

Streaming over HTTP/TCP

• Video file is fetched via standard HTTP GET requests over TCP.
• Server transmits at maximum rate permitted by TCP congestion control and flow control.
• In-order delivery and retransmissions introduce throughput fluctuations $x(t)$, requiring a larger client playout buffer.
• Basis for modern Dynamic Adaptive Streaming over HTTP (DASH), where video is chunked into small segments encoded at multiple bitrates, allowing the client to adaptively switch quality based on current TCP throughput.
• Advantages
  Easily passes through firewalls and middleboxes; leverages existing web infrastructure (CDNs, web caches).

Worked Example

Adaptive Streaming over a Variable Link

Adapted from December 2021, Question 2, December 2022, Question 2, November 2023, Question 2, and November 2024, Question 2.3.

Assume a stored video has aligned 2-second segments at 1, 2, and 4 Mbps. Segment boundaries support switching between representations. Ignore audio and transport overhead in the following sizes.

1. The server publishes a manifest listing representations and segment URLs.
2. The client fetches the manifest and estimates throughput from completed downloads.
3. At an estimated 3 Mbps, the client selects the 2 Mbps representation to leave a capacity margin.
4. Each 2-second segment arrives in approximately 1.33 seconds. Downloads can replenish the playout buffer faster than playback consumes it.
5. Throughput falls to 1.5 Mbps. A 2 Mbps segment now takes approximately 2.67 seconds, so continued requests at this quality drain the buffer.
6. The client requests subsequent segments at 1 Mbps. Each takes approximately 1.33 seconds at the new throughput.
7. The client increases quality only after throughput and buffer occupancy remain sufficient.

The buffer absorbs temporary deficits. Selecting a sustainable representation prevents a persistent deficit from exhausting it.

For live delivery:

• The encoder produces new segments continuously.
• The manifest advertises newly available segments.
• Shorter segments allow earlier delivery and faster quality changes, with more request and packaging overhead.
• Larger buffers reduce stalls but increase delay behind the live source.
• Interactive conversation requires a tighter latency budget than watching a live broadcast.