At the August 7, 2012 Peer Incite Research Meeting, Peak Colo’s Luke Norris laid out a blueprint for architecting modern networks. Essentially his organization had three main parameters in mind for the network architecture:
- Scale
- Simplicity
- High Availability
His strategy for achieving these results involved converging networks, virtualizing infrastructure, moving off spanning tree protocol (STP), and creating shared caching and solid state services.
Comparing legacy and modern networks, the former had separate infrastructure for voice (e.g. CAT 5), data networks (e.g. CAT 6) and storage (e.g. FC). Peak converged its networks onto Ethernet using NFS and iSCSI for the storage protocols such that all “data-ized” workloads run on a single converged data network.
Moving Off Spanning Tree
With legacy networks, organizations might have an FC SAN or older 1 gig Ethernet switches. This infrastructure used STP, which essentially builds a static map that dictates the data path. The data path is basically fixed causing bottlenecks and IO contention. This legacy approach is highly inefficient. For example, Norris indicated that older networks were over-provisioned at an 8:1 factor (to brute force around bottlenecks).
Moreover, to add a new switch in a legacy environment, the Spanning Tree map had to be re-drawn, which took time. Also, for many networks, adding capacity meant taking a planned outage – which could be avoided with engineering gymnastics, but was still onerous.
Peak moved to a TRILL fabric for its networks to resolve this problem. Peak actually uses a Brocade Virtual Cluster Switching (VCS) solution, which is TRILL-like. TRILL (Transparent Interconnect of Lots of Links) replaces spanning tree and enables fabric technology to be implemented in a mesh system that allows multiple paths between switches. This approach simplifies the network infrastructure, designs in scale and makes adding nodes non-disruptive. As such high availability networks are much easier to deploy.
Peak has also deployed flash virtually everywhere, in the server with SSDs and with Fusion-io cards and in the storage array using NetApp Flash Cache (and potentially Flash Pools in the future).
The result of this integration is a simplified infrastructure that is much more flexible and cost effective. As a proof point, the network switches today at Peak are overprovisioned only by a factor of 2:1 versus as much as 8:1 in legacy systems. Norris believes this infrastructure will carry the service provider into the next generation of cloud applications and support a business model that designs in the flexibility on which customers can build more robust solutions.
Action Item: New architectural thinking is required to design modern networks for scale. Specifically, CTOs should understand their technical objectives and design networks with low latency in mind and reduced bottlenecks. A key enabler of this objective is moving off the Spanning Tree protocol to a TRILL- or mesh-based system that enables any-to-any connectivity and more agile change management.
Footnotes: