What is I/O Virtualization?

Market Dynamics

I/O Virtualization is an emerging category of technologies aimed at resolving the performance, provisioning, and management problems of server I/O in highly virtualized and clustered data center environments. In the evolution of the virtualized data center, the issues of CPU and storage virtualization are actively being addressed, but are raising problems with the load, provisioning and management of storage and network infrastructure.

The Technology

It started with virtual memory, then virtual machines (CPUs), then virtual storage, and now I/O virtualization (IOV) – where the I/O path from the server to the peripheral is itself virtualized. Traditionally, I/O devices connect to the server with an interface or adapter — for instance, NIC – Network Interface Card, HBA – host bus adapter, etc. — which are located inside the physical server.

I/O virtualization moves the adapters out of the server and into a switching box. This allows the adapters to be shared across many physical servers, which drives up adapter utilization – often less than 10%-15% in a non-virtualized world. Fewer adapters means less power and cooling. Also, adapters take up a lot of space in servers, and moving them out of the server allows 1U servers to be used instead of 2U ones.

Also, by offloading many of the Hypervisor tasks from the host software to the adapter, virtualized applications can achieve performance levels previously experienced only on servers running in non-virtualized or native OS modes.

Moreover, depending on the vendor, the application server only needs to be touched once with the installation of one or more high-bandwidth IOV adapters. Changes to the network technologies – say moving from 4Gb Fibre Channel to 8Gb Fibre Channel — happen only at the switching box. What’s more, different I/O protocols can be carried by a single IOV, adapter further reducing complexity, disruption, power, and cost.

Standards

On the standards front, in June 2008, the PCI special interest group (PCI-SIG) announced it had completed a suite of I/O Virtualization specifications that in conjunction with system virtualization technologies allow multiple operating systems running simultaneously within a single computer to natively share PCI Express devices. These specifications are grouped into three areas:

• Address Translation Services (ATS) provides a set of transactions for PCI Express components to exchange and use translated addresses in support of native IOV. This essentially defines how many personalities (I/O types) can be presented.
• Single Root IOV (SR-IOV) provides native IOV in existing PCI Express topologies with a single root complex, e.g., a single server.
• Multi-Root IOV (MR-IOV) builds on the SR-IOV specification to provide native IOV in new topologies (such as blade servers) where multiple root complexes share a PCI Express hierarchy, e.g., multiple servers sharing a IOV switch.

This suite of standards has been leveraged many of the IOV vendors who were showcased at VMworld 2009 and Intel Development Forum 2009.

A Hybrid Approach

Xsigo – uses Infiniband Host Channel Adaptors and the company’s software on real machines to present virtual adaptors (vNICs and vHBAs) to VMs. It transports all I/O traffic to its I/O Director, where it simultaneously switches TCP/IP/Ethernet, Fibre Channel, and Infiniband to real I/O devices. Most notably, VMware embraced Xsigo at VMworld. Every server in its impressive 32-rack data center and every server in its booth used Xsigo virtual I/O. Xsigo was early to market and deserves credit for its wins so far.

Mellanox – This Infiniband supplier has also developed its server-edge gateway product, BridgeX, which supports both a 40Gbit/s Infiniband link to servers or a 10GigE link. Mellanox also just demonstrated a 40GbE adapter card that supports SR-IOV. Xsigo utilizes Mellanox technology

PCIe Extension

NextIO - The first company to develop IOV solutions using PCI Express, NextIO uses an inexpensive card and cable to simply extend a server’s PCIe bus to NextIO’s switching box rather than using an IOV adapter. NextIO is focused in several key vertical solutions and has partnered with several leading industry companies, including: IBM, nVidia, Dell, and Marvell. At VMworld, it was demonstrating a solid-state disk inside its switching box to help with high performance applications.

VirtenSys – Like NextIO, VirtenSys also extends the PCI bus with an inexpensive card and cable. It only announced availability of its IOV switches in August 2009 but won the Best Technology Award at VMworld. It is the first vendor to market to consolidate, virtualize, and share all the major types of server networking and storage connectivity, including Ethernet, Fibre Channel over Ethernet (FCoE), SAS/SATA, and Fibre Channel, without requiring any changes to the servers, networks, or I/O adapters. It also claims to be the most scalable, and it has some Xyratex heritage

Aprius – Though still under development, when complete, Aprius Gateway Systems will use an extended PCIe bus to virtualize and provide a unified access layer to a common pool of PCI Express-based I/O adapters including Ethernet, Converged Enhanced Ethernet (CEE), iSCSI, and FCoE.

Intel and AMD

Intel – Under its umbrella Virtualization Technology (VT), Intel’s Virtualization Technology for Directed I/O (VT-d) includes four key capabilities:

• I/O device assignment. This feature allows an administrator to assign I/O devices to virtual machines (VMs) in any desired configuration.
• DMA remapping supports address translations for device DMA data transfers.
• Interrupt remapping provides VM routing and isolation of device interrupts.
• Reliability features that report and record system software DMA and interrupt errors that may otherwise corrupt memory or impact VM isolation.

AMD – Just announced support for IOV with its new Fiorano and Kroner platforms that include multiple NorthBridges and advanced I/O Memory Management Units (IOMMUs).

Virtual NICs

QLogic – Provides its dual port 10-Gigabit PCIe Gen2.0 Intelligent Ethernet Adapter that supports SR-IOV and complete TCP/IP offload. Here, QLogic is leveraging its recent acquisition of NetXen.

Neterion - enables I/O virtualization of network traffic with 10 Gigabit Ethernet controllers. It recently demonstrated SR-IOV, which leveraged Intel’s VT-d, Red Hat’s Enterprise Linux Release 5.4’s advanced virtualization capabilities, and the company’s X3100 10GbE adapter.

Together with Intel and Red Hat, Neterion collaborated and integrated their platform, software, and hardware – Intel’s Xeon 5500 platform, Red Hat’s Enterprise Linux virtualization solution and X3100 10 GbE adapter – together, to build an end-to-end I/O virtualization solution stack. This was demonstrated at the recent Intel Developers Forum (IDF) 2009 in San Francisco

Dell - Dell recently demonstrated a SR-IOV solution using technologies from Intel and Citrix to help improve virtualization performance across networked servers and storage.

The solution utilizes Intel VT-d and the Intel® 82599 10 Gigabit Ethernet (10 GE) device with support for PCI-SIG SR-IOV specification and to allow multiple virtual machines (VMs) running on Citrix XenServer in a Dell PowerEdge R710 with Intel Xeon processor 5500 to directly share I/O devices. Dell and Intel worked closely with Citrix to develop the solution.

Solarflare Communications Solarstorm 10-Gigabit Ethernet server controller chips support a large number of vNICs, virtual PCIe functions, hypervisor cut-through and SR-IOV.

Blade Network Technologies offers its VMSmartConnect™ with VMready. VMready is currently available as part of the SmartConnect software upgrade for embedded Ethernet switches on IBM BladeCenter and HP BladeSystem.

SAN from DAS

LSI -- in collaboration with Intel and Red Hat recently demonstrated its SR-IOV capable storage controller. The controller, using Intel VT-d technology, enables virtual servers in direct-attached storage (DAS) environments to share a storage controller, delivering increased system performance, efficiency and scalability – a network-less SAN. Interestingly, LSI had written about this on Wikibon in 2008. Part 1 here and Part2 here.

Top of Rack Attack

Many the IOV vendors marketing pitches suggest there is a battle to be fought at the “top of the rack” where Cisco switches typically reside. They hope to win this battle with IOV products that deliver more function at cost less than Cisco. Certainly, the advent of technologies like FCoE demonstrates that users want to preserve infrastructure investments while evolving their data centers and not revolutionizing them. IOV lets users move in baby steps. As such, it has a future, but it is not clear yet just how long that will be.

Action Item: Users should certainly familiarize themselves with these emerging technologies and their potential cost savings, higher asset utilization, and performance improvements. It is clear that I/O is going to be virtualized and that it may happen rather quickly.

Footnotes: