SDN Test Suite – Motivation

This is the second in a series of posts about the SDN test suite that we’re building as part of the SDN Hub open-source ecosystem. Read a teaser here and the followup post with test results here.

Our goal is to build a modular, easily-configurable software package to test the scale, performance and functionality of SDN solutions; irrespective of whether the deployments is greenfield or brownfield, commercial or open-source, overlay-based or underlay-based, OpenFlow-based or proprietary API.

What is our primary driver?

It is an exciting time in the networking world, when the first killer application of Software-defined Networking (SDN) — Network Virtualization — is starting to be commercially available in the market. The key value proposition of SDN, in this context, is to enable the provisioning and operation of virtual networks and its connection to the legacy world (that includes the Internet, L4-L7 appliances, and bare-metal servers).

Screen Shot 2013-12-11 at 7.37.09 PM

There has been somewhat of an explosion in this solution space, which we’ve written about in detail here. As the adjacent matrix indicates, there are several classes of solutions from both the usual suspects, such as Cisco 9K series and VMware NSX, as well as new entrants into the market, such as PLUMgrid IOVisor and Big Switch BSN. The data plane can be overlay-based, or underlay-based, or combined. The control plane API can be OpenFlow, or an extended version of it, or a proprietary one.

Although there are choices available from several SDN vendors, the cloud service providers and enterprises have a hard time in selecting the solution that is right for their environment. The four main challenges in this selection are 1) the unclear differentiation between each solution, 2) the lack of standardized methodologies to evaluate the different offerings, 3) task force that is not ready for the new technology, and 4) the need for resources to evaluate solutions.

We, along with Nick Lippis and ONUG, recognized these challenges early on and endeavored to make the SDN adoption easier by building a software suite customized for testing the key value propositions.

Why do we need another Test Suite?

There are many testing tools, products, and vendors in the market already, so why have one more? Indeed, Ixia has the IxVM virtual appliance  which can test the end-to-end performance of various hardware or software network components. While solutions like IxVM are feature-rich, there are several shortcomings with conducting SDN tests using a virtual appliance.

First, SDN solutions have a large number of potential bottlenecks and touchpoints in a virtualized ecosystem. Here’s an incomplete listing that could affect application performance:

  • Type of virtualization technology (VMware, KVM, HyperV, Xen …)
  • Virtual switch/bridge technology
  • Number of CPU cores allocated to the VM
  • Number of CPU cores allocated to the virtual switch and its processing (e.g., encap/decap for overlay-based network virtualization)
  • The configuration parameters of the physical and virtual switches, routers, and gateways along the test path
  • The type and load of the traffic that is used to test

As convenient as it is to spin up a virtual test appliance, a appliance cannot capture many of these variables from the virtualized context, which is why we need to measure metrics from both the VM and the hypervisor to see the full picture.

Second, a generic blackbox virtual testing appliance cannot capture unique features introduced by SDN-based solutions. The control-data plane separation and the addition of a controller cluster adds an additional bottleneck that may have its own chokepoints. To test these, the testing setup must tie in closely with the SDN solution under test.

Third, most of today’s SDN testing and interoperabilty applies only to OpenFlow as managed by the ONF. Unfortunately, that doesn’t cut it for most users — The industry and the user groups are using solutions that are architecturally diverse. With the exception of a few players (like NEC and HP), the predominant network virtualization solutions do not use OpenFlow.

Fourth, and most importantly, we did not find an open-source, feature-rich, easily configurable test framework that could be dropped in to a large variety of SDN deployments. Much of the innovation in networking today is driven by openness and the open-source model, and it makes little sense to not allow the community to fork or contribute to an SDN-based project.

Our ultimate goal is to publish source code and test results of SDN solutions, both to demystify solutions and shed light on the tradeoffs of each solution, and also to encourage vendors and customers to contribute to the test suite and publish their own test results.

Interested in the Test Suite? Here’s how to help

If you are a customer, help us evangelize the test suite: use it to test your SDN deployment, recommend it to other users, contribute test specifications, contribute test results of SDN systems you use, or even contribute code. All of this will encourage further openness.

If you are a vendor, help us by making it easy to deploy or evaluate your solutions. Help us by removing restrictions on testing or publishing test results of your proprietary solutions: there are many truly innovative products and solutions out there and only an open apples-apples comparison will bring out the best features of your solution. Finally, also help us help you improve your products. For example, as a result of these tests, we have discovered bugs with the Juniper OpenContrail platform which the team can now fix so that it will not break in production.

Comments are closed.