David Vellante with Blair Parkhill
Anticipating that Exchange customers will increase mailbox capacity quotas when they upgrade to Exchange 2007, HP's Customer Focused Testing (CFT) Group last year initiated a project to identify the best way to reduce backup windows in Exchange 2007 environments. Because users will have more data to protect with no more time to perform backups, they will need to find ways to reliably accelerate backup processes.
The basic premise of this Peer Incite is that by taking advantage of HP’s testing efforts and leveraging its recommendations, Exchange customers can make better technology choices for their specific environments, optimize backup performance, speed implementation of new processes, cut costs, and reduce implementation risks.
The Project HP configured Microsoft Exchange 2007 using HP Proliant BladeServers and the HP EVA8000 as the primary Exchange data store. The HP EVA6000 was used as the disk-to-disk target array and was configured with FATA drives. An HP ESL 712e tape library was used as dedicated backup and restore device. HP tested NTBackup as the backup application.
The EVA8000 used 146 300GB 10K rpm FC drives and the EVA6000 used 80 500GB 10K FATA drives. The ESL tape library was used as a nearline storage device and was configured with 16 HP LTO3 FC tape drives. The system tested two Exchange 2007 mailbox servers, the first hosting 5,000 users with 100MB mailboxes, and the second, 5,000 users with 1GB mailboxes.
HP ran a number of tests to determine the effect of concurrency on backup performance. Specifically, HP tested various combinations of backup using X number of concurrent streams going to Y number of disks or tape drives—ranging, including 4 concurrent streams to 4 disks, 4 concurrent streams to 4 tape drives, 32 concurrent streams to 32 disks, and 10 concurrent streams to 10 tape drives and other configurations.
HP wanted to understand the core differences in the streaming API for Exchange 2003 versus Exchange 2007 and determine the degree to which Exchange 2007 enhancements could be exploited, namely the ability to support more storage groups and hence greater concurrency during backups.
Findings
What are the most effective techniques to reduce Exchange backup windows?
HP testing found that in general, using more storage groups in Exchange 2007 will result in greater backup concurrency, which should provide faster backups. However increasing the number of storage groups can also increase complexity for storage administrators and in some cases degrade performance if server kernel memory becomes overtaxed. HP found that reducing the number of storage partitions (e.g. sending multiple streams to a single disk volume) can be an effective backup technique without degrading performance and without the need to interleave data, which would negatively impact restore performance.
In addition, when imposing the storage group limitations of Exchange 2003 (i.e. four storage groups), HP found that tape devices were more efficient and could achieve higher transfer rates than disk-to-disk backup. Specifically, four concurrent streams to four tape devices using hardware compression and data buffering allowed the backup of a 520GB database to be completed in 1 hour and 30 minutes versus 1 hour and 50 minutes using four streams to four disk devices.
On balance, however, increasing the number of storage groups and concurrent streams and using disk-to-disk backup yields the best performance and reliability for NTBackup. The improvements are not linear, however, and users should be aware that increasing the number of storage groups and concurrent streams can tax server resources and increase complexity. Backup Executive, DataProtector, and NetBackup users may find better performance with tape.
What about Volume Shadow Copy Service (VSS)?
VSS allows a point-in-time copy of open files and databases. Microsoft has improved VSS in support of Exchange 2007 and simplified its use, although users should be cautious, because VSS uses a serial method and concurrent backups can be problematic if certain criteria are not met.
VSS can offer options to users looking for continuous or near continuous backup solutions. Also, VSS can allow for passive database backup where a replica of the active database can be created and backed up on a separate set of disks, reducing contention for server and storage resources.
For applications requiring consistent point-in-time copies, VSS is pointing the way in which servers, storage and applications interact in Exchange environments.
What are the tradeoffs between using disk- versus tape-based backup in Exchange?
The most important advantage of disk-to-disk backup is it can be initiated with multiple streams to the same target without spreading data across multiple pieces of media. This takes advantage of disk’s random access methodology and keeps recovery times acceptable. Disk-based backup, however, introduces substantial server overhead due to file systems management needs. Memory and CPU overhead can also be significant in large backup scenarios.
The main advantage of tape-based backup is it remains the fastest elapsed time technique. If RPO is critical, put in lots of tape drives and ignore the costs. With multiple tape drives, recovery is enhanced due to the multiple streams.
Advice for Administrators
Critical best practices that emerged from this project include:
- Larger is not necessarily better-- keep Exchange database files to manageable sizes (e.g. around 25-50GB if possible) and use more storage groups and volumes.
- Backup disk array configuration data. This will accelerate restoration in the event the array needs to be replaced.
- Watch event logs. Windows event logs provide excellent visibility into activities such as log truncation and general backup health.
- Monitor server workloads to ensure that the backup job is not overtaxing server resources—keep page pool memory below 180MB for optimal performance and efficiency.
Action Item: Exchange 2007 customers should rethink backup and restore processes and take advantage of support for increased concurrency and simplified point-in-time copy facilities. In general, the more concurrent streams made available during backup, the faster backups will perform. However, users should balance these benefits with cost and complexity of increased concurrency and storage group management.
Footnotes:
