The traditional 3-tier (access/aggregation/core) architecture is undergoing a transformation due to the adoption of server virtualization and higher speed Ethernet (10GbE and higher). The general term for the new architectures are fabrics – a flatter architecture that looks to overcome the limitations of Spanning Tree Protocol (STP). Each networking vendor has different terminology and often multiple solutions to transform networking design. Standards implications include: Layer 2 Multipath (L2MP) alternatives are being worked on by IETF (TRILL) and IEEE (SPB) groups. An alternative to changing L2 to accommodate the new requirements is L3 routing. Below are some of the considerations that distinguish the various offerings and some of the criteria on which to decide among the different architectural decisions. Note that this discussion is orthogonal to FCoE; L2MP is not a prerequisite of multi-hop FCoE, especially if routing from FCF to FCF.

Edits and comments to clarify the technology are welcome; please keep vendor-specific comparisons to a minimum.

Layer 2 Multipath

Scalability

STP limits connectivity to a single connection for each device. As 10Gb Ethernet is adopted and devices demand more data, inactive links become much more expensive and inefficient. Data center architects typically will limit the scalability of a L2 environment and use vLANs to limit performance issues from broadcasts or reliability issues from failures.

Virtualization

Virtualization increases the bandwidth between servers (edge-to-edge rather than edge-to-core – see a from Greg Ferro, aka Etherealmind, provides a good discussion with diagrams.) vMotion is limited to L2 (see Ivan Pepelnjak aka IOSHints discussion of why not L3). In general, vMotion is typically done in very local areas (first uses were typically between blades of a blade server or servers in a rack) inside of a cluster which can be contained to a single L2 domain. vMotion between clusters or even between data centers while possible, requires careful implementation and should not be the primary driver for a network architecture.

Status of the standards

IETF (TRILL) was ratified in July 2011 (RFC 5556), while IEEE (SPB) is not yet a ratified standard yet, but is generally complete from a technology standpoint as they work through the machinations of the standards process.

Vendor solutions

• Cisco has stated that TRILL will be supported when the standard is done, in addition they are shipping FabricPath, which they consider a superset of TRILL.
• Brocade VCS includes both DCB and data plane TRILL functionality. They use FSPF instead of IS-IS in the control plane, making their current implementation of VCS fabric proprietary. Brocade does plan to support standards-based TRILL in the future.
• HP will support both SPB and TRILL (see this HP blog post).
• Juniper QFabric is architected as a “single tier”, so does not require L2MP between what is traditionally the edge and core.
• Dell/Force10 Open Cloud Networking includes FTOS advanced software features including TRILL (Transparent Interconnection of Lots of Links), DCB (Data Center Bridging), EVB (Ethernet Virtual Bridging), and VEPA (Virtual Ethernet Port Aggregation).

Layer 3

Layer 3 functionality is supported in most core switches and many edge switches. Using L3 for scalability allows administrators to limit L2 domains to small islands that will not require L2MP. As noted above, vMotion is limited to L2.

Vendor solutions

• Cisco Nexus 7000, Nexus 5548/5596 (earlier Nexus 5000 series switches are L2 only) and Nexus 3000 switches support L2/L3.
• Juniper QFX3500 supports L2/L3.
• Dell/Force10 switches support L2/L3.
• HP IRF is a combination of L2/L3.
• Arista Networks switches support L2/L3

What works best in your datacenter?

Just like the protocol arguments in the storage world, customers will need to choose the solution that works best for the scale, skill-set, equipment and applications for their environments. There are also implications of the choice to L4-L7 services, see this site from Intel/Cisco as a reference. Data centers are going to be a mix of L2, L3, next-gen-L2 for a long time.

Action Item: While every vendor is discussing the latest "fabric" technology, IT organizations are grappling with the ripple effects of virtualization on networks and determining how to roll out new architectures with 10Gb Ethernet (and watching 40Gb/100Gb Ethernet coming soon). CIOs and Network Architects should look for proof points of proven architectures that have been implemented by peers. Vendors need to focus on delivering best practices and services to help customers make an informed decision on designing a flexible network architecture.

Footnotes: More on Data Center Ethernet Fabrics