Fibre Channel over Ethernet (FCoE) is being hailed as a key new technology that is a first step towards consolidation of the Fibre Channel storage networks and Ethernet data networks into a single entity. This has several benefits including simplified network management, elimination of redundant cabling, switches, etc., reduced power and heat requirements, and enhanced data network performance, since FCoE is built on the new 10 Gbit lossless enhanced Ethernet (and should really be called Fibre Channel over Enhanced Ethernet, therefore or FCoEE).
Essentially FCoE encapsulates FC frames in Ethernet packets and uses Ethernet instead of Fibre Channel links. Nonetheless, at the upper layers it is still Fibre Channel, which supports the preservation of existing FC infrastructures – a major design goal. Storage management using an FCoE interface has the same look and feel as storage management with traditional FC interfaces.
One of the complexities of the FCoE picture is that the ecosystem of the standards span three separate standards bodies:
- INCITS/ANSI T11 committee for the Fibre Channel protocols,
- IEEE for Ethernet extensions,
- IETF for routing.
T11 creates FCoE, IEEE creates the lossless Ethernet environment. T11 approved the Fibre Channel Backbone 5 (INCITS/ANSI T11 FC-BB-5) in June, 2009, thus we can say that the FCoE standard is done. It defined the network and its associated resources and services that are used to connect one or more Fibre Channel entities over non-Fibre Channel protocol infrastructures. While the standard covers many use cases, FCoE, the focus here, is part of FC-BB-5 and is technically known as FC-BB_E. The FC-BB_E model defines the means by which Fibre Channel frames are transported over a lossless Ethernet network.
Lossless Ethernet includes new enhancements that make it a viable transport for storage traffic and storage fabrics without requiring TCP/IP overheads, incorporated at this time in a 10Gbps full duplex, lossless Ethernet with dynamic prioritization/bandwidth allocation and some other goodies. This is known variously as:
- CEE: Converged Enhanced Ethernet – a generic term used by many vendors including HP, IBM, Dell, and Brocade to describe enhanced Ethernet.
- DCE: Data Center Ethernet (term no longer in use)– a term originally coined and trademarked by Cisco. DCE refers to Enhanced Ethernet based on the Data Center Bridging standards and is CEE plus additional functionality including congestion notification
- EEDC: Enhanced Ethernet for Data Center (term no longer in use)– a term coined by Intel that also includes deterministic latency for high-performance computing and communications traffic as well as seamless integration with classical Ethernet networks.
- DCB: Data Center Bridging – a work-in-progress in an IEEE task group described as an architectural collection of Ethernet extensions designed to improved Ethernet networking and management in the data center.
All four of these terms are pretty much interchangeable in the context of this document.
A Lossless Ethernet environment can be created with flow control such as 802.3x PAUSE, but for FCoE is replaced with Priority Flow Control discussed below. Additional IEEE and IETF Ethernet Enhancements beyond creating the lossless environment are optional for FCoE environments and will vary by vendor. The DCB Ethernet enhancements are defined by the following IEEE specifications:
- 802.1Qbb: Priority Flow Control (PFC): passed sponsor ballot in June '10
- Provides the ability to control a flow (pause) based on a priority,
- Allows lossless FCoE traffic without affecting classical Ethernet traffic,
- Establishes priority groups using 802.1Q tags.
- 802.1Qaz: Enhanced Transmission Selection (ETS):
- Allows bandwidth allocation based on Priority Groups,
- Allows Strict Priority for low bandwidth / low latency traffic,
- Incorporates DCBX (DCB Exchange Notification) is part of 802.1Qaz and leverages LLDP (Link Layer Discovery Protocol, 802.1AB).
- 802.1Qau: Congestion Notification (CN)
- Allows for throttling of traffic at the edge of the network when congestion occurs within the network.
- 802.1aq: Shortest Path Bridging (SPB)
- Enables shortest path trees thereby providing the ability to use all available physical connectivity, because of loop avoidance.
These IEEE specifications are not final or published yet but are presently considered “technically stable.” This basically means they have been reviewed by many people and have had sufficient working demonstrations that no technical changes are expected and it is safe enough for vendors to implement. Possible changes, if any, will be minor and with little effect on implementations. Actual approval is expected sometime in 2010, depending on who you ask.
The IETF is working on a solution similar to SPB called TRILL (Transparent Interconnect of a Lot of Links) to enable Layer 2 Multipathing. Cisco's FabricPath solution is a "superset" of the TRILL functionality.
Action Item: FCoE is now a standard. Products produced by vendors in 2010 should meet the T11 and IEEE standards. Updates at IEEE should for the most part be supported in products through a software update. Interested organizations have started to build FCoE networks in test and production environments. Standards are no longer the limiting factor to deployments. Practitioners should speak with vendors to understand the full details of the configurations that can be deployed and look for guidance in the convergence of LAN and SAN.
Footnotes: IEEE 802.1 Data Center Bridging Task Group
