Crossbow on Big F#@!ing Webtone Switch
Back in the days of SUN Microsystem, Scott McNealy asked us to build a big F#@!ing Webtone Switch. At that time, the underlying pieces weren’t there but over last few years the possibilities have opened up. We now have the switch chips from Broadcom and Intel that switch at 1.2Tbps in H/W. From a OS view, 1.2 Tbps of switching at 300ns latency is great but the more amazing thing is PCIe as a control plane which allows 20-40Gbps of control plane B/W where you can change switch registers, L2/L3-tables, TCAMs, etc at nano-second rates.
So after more than three years of work and million lines of C code, the Pluribus Network’s engineering team has the switch chip under Crossbow control. For people who are not sure what I am talking about, in 2005 project Crossbow invented virtual switching inside a server hypervisor and introduced hardware based Virtual NICs and dynamic polling to get 40Gbps of bandwidth through a server OS. The details were published in “Crossbow: From Hardware Virtualized NICs to Virtualized Networks” in ACM Sigcomm VISA 09.
In the goal to benefit from merchent silicon ecosystem and orchestrate the entire infrastructure using Open source OS on switches, the industry has been going on suboptimal paths. The most notable efforts around a centralized controller can barely deal with the scale of single switch and typically requires sending a packet to a controller running on a separate server. The latency of these transactions (typically in milliseconds to seconds) defeats the required reaction time in microseconds in virtualized environments where Network resources are shared. The other approach of just throwing the Intel or Broadcom SDK on a whitebox switch with Linux and Quagga doesn’t really solve the control plane problem. The Broadcom and Intel SDK are crafted for their specific switch chips and meant for configuration ease and not for high speed control plane software.
By bringing the Crossbow Architecture on the switch chip where it is part of the Network OS directly controlling the switch chip via the PCIe allowsus to get following benefit:
- Integrated Switch Hardware with fully programmable Control Plane allowing the performance and scale necessary to deal with 10Gbps switches (the distributed control plane is part of the Network OS running on the switch itself).
- Enable applications like DDoS, IDS, Firewall, Load Balancer, routing, messaging, etc that need to be in network to run on the switch itself and benefit from the H/W offload, high speed snooping, and flow capability that switch chip offers via C, Java, Perl, Python, etc programming interfaces in UNIX/Linux environment. Development, Deployment and Resource provisioning of these applications on Crossbow enabled switches is same as current server mechanisms and uses the existing tool chain (gcc/gdb, kvm, etc).
- Bring the benefit of merchent silicon ecosystem on switches under Openstack control enabling faster pace of innovation and cost advantages.
As we get ready to roll Netvisor (and its open source version – openNetvisor) out, I will discuss more details on this blog in near future.