John Ousterhout, Stanford Professor and author of A Philosophy of Software Design, discusses the evolving nature of AI networking workloads and why traditional protocols like TCP and RDMA are becoming bottlenecks in modern data center environments.
The Shift in Workloads
- Historical context: AI traffic was dominated by massive, long-running transfers (gigabytes of gradients), where throughput was the primary metric (2:19-2:42).
- Modern AI: Workloads, especially inference and agentic applications, now rely on frequent, small coordination messages (e.g., KV cache lookups, barrier synchronization). These small messages are highly sensitive to latency (3:09-4:02).
- The Bottleneck: When small synchronization messages are mixed with large traffic, they get trapped in queues (caused by incast), significantly increasing 99th percentile (tail) latency. This causes GPUs to sit idle, wasting expensive compute resources (4:24-5:34).
Why Legacy Protocols Struggle
- Sender-Driven Congestion Control: TCP and RDMA rely on the sender to detect congestion, often via packet drops or delayed signals from switches. This process is inherently reactive and oscillates, leading to unstable performance (7:10-9:58).
- Byte Stream Model: These protocols view data as an opaque stream of bytes rather than discrete messages, making it difficult to prioritize short, critical tasks (10:05-11:05).
The Homa Solution
John introduces Homa, a clean-slate transport protocol designed for data centers (11:15-12:15):
- Message-Based: Unlike byte streams, Homa understands message boundaries, allowing it to predict traffic and prioritize short messages using Shortest Remaining Processing Time (SRPT) (12:22-13:28).
- Receiver-Driven: The receiver controls the flow by issuing grants to senders, effectively managing congestion before it occurs at the switch (13:30-14:58).
- Priority Queues: Homa leverages the multiple hardware queues already present in modern switches to bypass long, queued traffic with low-latency short messages (14:59-15:51).
- Performance Results: Benchmarks show Homa can reduce tail latency for short messages by over 10x compared to TCP, while simultaneously improving performance for large messages (15:52-17:27).
John Ousterhout, Stanford Professor Emeritus, discusses the changing nature of AI workloads, emphasizing the increasing importance of low latency for small message exchanges, particularly in inference and agentic applications. He explains how traditional protocols like TCP and RDMA, designed for high throughput of large data transfers, struggle with these new latency-sensitive workloads. This is often due to sender-driven congestion control and a byte-stream model that cannot prioritize short messages, leading to high tail latency when mixed with large traffic.
Ousterhout introduces Homa, a new transport protocol designed from scratch for data centers. Homa is message-based, allowing it to understand message boundaries and prioritize shorter messages using SRPT. It employs a receiver-driven congestion control mechanism, giving the receiver more precise control over flow and congestion. Furthermore, Homa utilizes the priority queues available in modern network switches to ensure short messages bypass longer, queued traffic.
Benchmarks presented show Homa significantly reduces tail latency for short messages (over 10x improvement compared to TCP) while also maintaining or improving performance for larger messages. Ousterhout highlights that Homa is his current primary focus, and he is actively working to make it production-ready, offering support for those interested in experimenting with it.
Speaker Info:
Timestamps: 0:00 - Why latency is becoming the metric that matters 2:19 - The old workload: gigabytes and throughput 3:09 - The new workload: metadata and coordination 4:24 - How one slow exchange stalls every GPU 6:07 - Incast, and where the queue actually builds 7:10 - Why congestion control lives on the wrong end 10:05 - A byte stream has no message boundaries 11:08 - Homa, and a clean slate redesign 12:12 - Messages, not streams 13:30 - Controlling congestion from the receiver 14:59 - Using the priority queues already in the switch 15:52 - The benchmark against TCP
