Datacenter networks have evolved over the years in both scale and complexity of the end-to-end topology. Large datacenter networks serving highly volatile workloads are deployed using multi-tier Clos topologies with multiple paths between end-points to scale bandwidth. In addition, virtual L2/L3 topologies with a rich set of L4-L7 services can be built rapidly in seconds to connect physical servers, virtual machines and containers. Traditional network monitoring techniques based on fetching state from individual network elements through the control plane are either too restrictive or too slow to meet the needs of such high-scale networks with virtual and physical layers and advanced network functions.
We present a framework called “In-band Network Telemetry” (INT) that enables collection of end-to-end real-time state directly in the datapath. A source end-point embeds instructions in packets listing the types of network state to be collected from the network elements. Each network element inserts the requested network state in the packet as it traverses the network. P4 provides a natural way to express the kind of packet header parsing and modifications required for INT.
Ability to collect real-time end-to-end network state opens up many new possibilities. We present two main use cases for INT. First, we present the application of INT to network monitoring and troubleshooting. Second, we present novel algorithms that use network state collected at RTT timescales to achieve effective load balancing of traffic across equal-cost paths, translating to better performance for both throughput-heavy and latency-sensitive applications.
