I/O Virtualization Gets Interesting

Just as IT organizations begin to get a handle on server and storage virtualization, along comes yet another "V" category: I/O virtualization. Some of you are early adopters, using products such as the Virtual Connect functions in HP's blades or 3rd-party systems from companies such as Xsigo or 3Leaf. In general these solutions do reduce complexity when trying to manage the unruly forest of physical connections that go into a rack of virtual server or blade hosts. But these first generation solutions suffer from a number of problems including:

• Proprietary implementations: every blade vendor has their own solution with a small ecosystem of partners who produce add-on products. For 3rd-party suppliers the problem is similar, i.e. select one of those 3rd-party solutions and you are wedded to it for good or ill, so best hope it works and the vendor continues to support it.
• Scalability issues: I/O virtualization does nothing to help with general I/O scalability, in fact some implementations that place an I/O virtualization controller in-band between the host and the I/O resources can make things worse.
• Limited functionality: Existing solutions can only hide the fact that the hardware doesn't know how to share access to a device. So VMs have to go through the hypervisor to access the device. In addition, there is no way to share one device between multiple hosts or move the interface to a new host if the original host goes down.

So these limited I/O virtualization schemes maybe useful in the short term, but in the long term we need something better because organizations will build larger, more mission critical virtual server environments emphasizing the need for more flexible I/O management. Complex clusters of servers supporting server virtualization will inevitably have many connections to SAN and network resources, and the last thing you want to do in the middle of a failure is start messing around with network configurations, SAN zoning, or arrays, and virtualizing the I/O connections is the only way to avoid this. But it's not just about connections, it's also about:

• More intelligent hardware: Offload the virtualization of interfaces and connections from the hypervisor to an intelligent adapter that can do the heavy lifting when it comes to sharing a device between multiple hosts and VMs.
• More scalable I/O options: Moving the I/O slots outside of the server enclosure will allow small form factor servers and blades to reduce the stark trade off between I/O capabilities and physical density in future data centers.
• Industry standards-based solutions: If there is one thing that accounts for the success of the x86 platform, it's commoditization through a standard hardware platform. I/O virtualization needs a standard hardware platform that level the I/O virtualization playing field so that anybody can play, shifting the emphasis onto software to manage the virtual infrastructure.

This may sound a bit "blue sky" but it's a critical step on the road to the dynamic data center, and is actually closer to reality than you might think. Last week's announcement from the PCI SIG of a complete multi-root I/O virtualization spec (MR-IOV) specification being the latest important step. To misquote Churchill, 'MR-IOV doesn't signal the end of conventional PCI implementations, not even the beginning of the end, but it is the end of the beginning'¹. The industry now has in place the standards needed to revolutionize PCI I/O:

• Single root I/O virtualization (SR-IOV): SR-IOV allows an intelligent adapter to create multiple virtual adapters for the physical host. This virtual adapters can be assigned directly to a VM instead of relying on the hypervisor to arbitrate everything. SR-IOV products are already shipping from companies like Neterion.

SR-IOV adapter with multiple virtual hardware interfaces assigned to virtual machines

SR-IOV is only half the solution, because the interface is limited to one physical host, which creates some additional problems, particularly with fail over of VMs where identical hardware on the new host will be required.

• Multi root I/O virtualization (MR-IOV): MR-IOV creates a switched PCIe infrastructure that can extend the I/O capabilities of the server in three ways:

MR-IOV configuration with shared virtualized I/O interfaces

1. Inserting a switched infrastructure allows the server to be connected to additional I/O slots, which gets around the form factor limitations of rack and 1U/2U servers.
2. With a intelligent adapter that can create multiple virtual hardware interfaces, adapters can be shared between physical hosts, making more efficient use of high bandwidth interfaces.
3. Because adapters can be seen by multiple physical hosts with actual assignment to hosts handled in software, an adapter can fail over to a new host during as part of a cluster fail over event.

The capabilities delivered by MR- and SR-IOV allows us to completely decouple the physical host from the I/O slots. Ultimately we may end up with a server that has no conventional PCIe slots, instead it has PCIe connections (and you could fit several of those in the space of one PCIe slot) to a shared, switched I/O subsystem. Configuration of devices, and their assignment to physical hosts will be dynamic and managed through provisioning and orchestration software. The leader in bringing MR-IOV to market is NextIO which already working with companies like IBM and Neterion to deliver a fully virtualized I/O subsystem to a data center near you.

The long term benefit of this approach over any of the existing solutions is that it will become a standard part of the commodity server platform which will encourage the development of intelligent adapters and the management/provisioning software needed to manage a virtualized I/O infrastructure.

Posted by: Nik Simpson