The purpose of this how-to article is to hash out an open standard benchmarking system for storage devices. Storage devices are complicated, and this makes it hard to compare one device to another. Unlike systems and servers (which are benchmarked in a fairly straightforward way), storage devices have multiple components that contribute performance (and performance limitations) to different types of applications at different points.
Contents |
Definitions
To start any discourse on benchmarking, a common pool of definitions is needed.
Storage Device
storage device definition
Host device
host device definition (operating system, applications)
Anatomy of I/O
Different applications perform input and output differently.
Sequential
sequential
Random
random
Read/Write Ratio
Read/Write Ratio
Protocols
List of protocols
FC
FC FICON
Network
iSCSI file protocols
Direct attached
- note: there is a convergence between direct attached and networked protocols
SCSI ESCON SAS
Benchmarking
Benchmarking storage is complicated. In a more linear system with less complexity, benchmarks are
Storage Benchmarking History
specifically SPC
SPC
history, background, controversy
Pros
only benchmark available currently, allows companies to see if/when the device they're considering will no longer be able to saturate the back end disks
Cons and Common Criticisms
not reflective of real life, does not reflect anything but back end device performance
Storage Benchmarking Future
Proposed new open benchmarks
Usefulness
Should provide as close to real life comparison as possible, if necessary with many different workload simulators for different types of environments
Neutrality
Should be tested by multiple parties, the more the better. At least one should be either neutral or a competitor to the vendor (client or competitor)
Openness
Should allow anyone to recreate the test with the same gear to check the reproducibility of the test, should also allow anyone to create a variant or fork