On September 28 2011, Oracle CEO Larry Ellison approved the press release “Another Whopper from Autonomy CEO Mike Lynch”. Ellison is right in calling out this lapse in ethical standards.

However, two days beforehand it appears that Larry and/or the team advising him may have been more than a little economical with the truth with the performance claims made during the Oracle webcast announcement of the new SPARC servers. Larry claimed nine world record benchmarks with SPARC T4-4 servers and continually claimed that the SPARC-based systems have twice the performance of IBM and HP servers, at one-third or one-quarter of the price.

The repeat performance on Sunday’s Oracle OpenWorld still emphasized the performance improvements (while admitting that IBM’s system was faster with arithmetic). There was less emphasis on world records. The Oracle press release calls them “Record-Breaking Performance Results”, a step down from world records.

Two of the tests were industry-standard benchmarks. They were:-

• TPC-H @ 1000GB,
• SPECjEnterprise2010.

Seven of the tests were Oracle internal benchmarks:-

• Oracle PeopleSoft HRMS Self-Service 9.1,
• Oracle PeopleSoft Enterprise Payroll (N.A.) 9.1,
• The Oracle E-Business Suite R12.1.2 XL,
• The Oracle E-Business Suite R12.1.2 XL,
• Oracle Siebel Loyalty Management,
• Oracle Communications Applications,
• Cryptography.

Wikibon asserts that there was only one meaningful world record, and that was the way in which the benchmark data was interpreted incorrectly by Oracle. Wikibon’s analysis of the same evidence presented by Oracle is that the Oracle/Sun and IBM RISC systems are about the same performance and price/performance, and there is no evidence to assess the performance of the HP systems against Oracle.

Deep Dive on the TPC-H Results

The Oracle information about the TPC-H result was as follows:

“On this industry-standard benchmark that simulates a decision support system providing answers to critical business questions and consists of a suite of business oriented queries and data with broad industry-wide relevance, the SPARC T4-4 server, with four SPARC T4 3 GHz processors, running Oracle Database 11g Release 2 and Oracle Solaris, delivered over 2.4 times better performance per processor at 1/3 the price/performance than an eight processor IBM Power 780, and 44 percent faster performance at 2.6 times better price/performance than a 16 processor HP Superdome 2 (2).”

TCPH Performance

Table 1 shows the TPCH results for 1,000 GB from the TCP website:

Table 1: Top Ten TCPH at 1,000GB Results as of 9/29/2011 including Clustered and Non-clustered Solutions
Source: tpc.org all TPCH at 1,000 GB results, downloaded 9/29/2011

At first glance, there does not seem to be any world record for Oracle; but wait, at 25 minutes 50 seconds into Larry Ellison’s presentation there is a slide which says “9 world records against IBM & HP”.

Could it be argued that the clustered systems in the first six spots in Table 1 are unfair and deserve a kind of steroid disqualification? Table 2 shows the top 10 un-clustered systems from the TPC website, and Oracle is in third place. Two more entries to eliminate before a world record can be claimed.

Table 2: Top Ten TCPH at 1,000GB Results as of 9/29/2011 for Non-clustered Solutions
Source: tpc.org non-clustered TPCH at 1,000 GB results, downloaded 9/29/2011

Maybe Oracle can argue that Dell and HP ProLiant TPCH results don’t count in the world record stakes? Is it reasonable to make an assumption that Linux and Windows are disqualified, and that Oracle has a world record over IBM and HP in the 1,000 Gigabyte TCPH category?

Table 3 shows a more detailed summary including other information from the TCPH submissions. The TPCH performance metric is QphH@database size. Sun's analysis uses this metric and divides it by the number of processors. TPCH does not use this “random” metric, and there is no technical reason to choose the Oracle metric in this particular benchmark situation.

Table 3: Wikibon Analysis Top Ten TCPH Results as of 9/29/2011 for Non-clustered Solutions
Source: tpc.org non-clustered TPCH 1,000 GB results, together with additional detail from individual submissions, downloaded 9/29/2011

The raw results in Table 3 show that the IBM system is 18% (100% - 82%) slower than the Oracle T4-4 server. They both have 32 cores, and if the configurations are similar it could be reasonable to conclude that for this benchmark workload the Oracle T4-4 is a little faster.

However, the configurations are not the same. The Oracle T4-4 has 2.5 times as much storage (as is shown in the Total Storage/Database Size Ratio in Table 3), which probably means there were a lot of short-stroked disks. From the point of view of a benchmark result, the 18% difference is valid. However, it fails completely from the point of view of providing a proof-point that the SPARC T4-4 server is over twice as powerful as the IBM Power 780 server.

The HP Superdome system was submitted 17 months ago and is not the latest Superdome system. The benchmark adds very little to the understanding of the relative performance of HP’s current Superdome systems and the SPARC T4-4.

Wikibon’s conclusion from the data available is that the IBM and SPARC systems are about equal in performance for the TCPH workload measured.

TCPH Price Performance

The raw price comparison in Table 4 shows that the SPARC server was 33% less expensive.

Table 4: Wikibon Analysis Top Ten TCPH Results as of 9/29/2011 for Non-clustered Solutions
Source: tpc.org non-clustered TPCH 1,000 GB results, together with additional detail from individual submissions, downloaded 9/29/2011

However, the IBM system was measured and priced 9½ months before the Oracle system. The natural price erosion from Moore’s Law over time is about 3.9%/month (50% over 18 months). Table 4 adds in the time adjustment factor, and calculates a corrected price/performance comparison. The last column in Table 4 shows that if you are concerned with using the benchmark figures to compare the current price performance of the different systems, the IBM system is 4 % higher than the SPARC systems.

The previous section pointed out that the storage systems of the IBM and Oracle systems were different and that if this factor were taken into consideration, the performance figures of the IBM and Oracle systems would be about the same. If that factor was added to the analysis in Table 4, together with the decrease of the cost of a smaller storage system, the price performance of the IBM system may be 5%-10% better than the SPARC T4-4.

The HP Superdome system was submitted 17 months ago, and is not the latest Superdome system. The benchmark adds very little to the understanding of the relative price performance of HP’s current Superdome systems and the SPARC T4-4.

Wikibon suggests that the best conclusion from the data available is that the IBM and SPARC systems are about equal in price/performance for the TCPH workload measured. Table 4 does not support in any way Larry Ellison’s contention that the SPARC server is one-quarter the price of the IBM server.

The TCPH Benchmark Bottom Line

The benchmark indicates that the IBM 780 and Oracle T4-4 systems are about equal in performance and price for this workload. The benchmark says almost nothing about the performance and price/performance of the current HP Superdome.

This analysis does not support in any way Oracle’s contention that the SPARC server is twice the performance and one-third the price of the IBM server. Wikibon believes that the price and price/performance data based on the TCPH figures presented by Oracle was deliberately misleading.

Deep Dive on the SPECjEnterprise2010™ Results

The SPECjEnterprise2010 benchmark is a full-system benchmark, which allows performance measurement and characterization of Java EE 5.0 servers and supporting infrastructure such as JVM, Database, CPU, disk, and servers. The SPECjEnterprise2010 is a much less well known benchmark than the TPC benchmarks. It only considers performance, not price/performance – this is explicitly spelt out in the benchmark description. SPEC does not endorse any price/performance metric for the SPECjEnterprise2010 benchmark but does not prohibit using the Bill of Material (BOM) to calculate price/performance. There have been 23 benchmark results submitted since its inception in December 2009. Fifteen of the 23 have been submitted by IBM. Oracle has submitted three benchmarks using Oracle hardware.

SPECjEnterprise2010 Price Performance

Table 5 shows the results of the benchmark from the 3Q section from the SPEC website (excluding a CISCO system). The results show that Oracle EjOPS result of 40,105 vs. an IBM result of 16,646. This is definitely a world record for the Oracle benchmark.

Table 5: Wikibon Table of SPECjEnterprise2010 benchmark results by Oracle and IBM as of 9/29/2011 for Non-clustered Solutions
Source: SPECjEnterprise2010 benchmarks for IBM and Oracle, downloaded 9/29/2011

One key question is looking at the implications of the benchmark is the performance of the servers. The quickest of glances at the table will show that the Oracle configuration is twice as big as the IBM configuration. That is fine if you want to achieve the highest performance on the benchmark. However, if you want to compare the servers, you have to:

• Compare the equivalent number of servers,
• Ensure that the software and hardware configuration are very similar.

Table 6 normalizes the data from Table 5 for cores and chips. Assuming that the configurations and software were roughly the same, it shows that the IBM server was 17% less powerful than the Oracle System.

Table 6: Wikibon Analysis of SPECjEnterprise2010 benchmarks by Oracle and IBM as of 9/29/2011
Source: Wikibon 2011 analysis of data from SPECjEnterprise2010 benchmarks for IBM and Oracle, downloaded 9/29/2011

However, as with the TCPH results above, the configurations are not the same. One huge difference is the inclusion of 8 Sun Storage F5100 Flash Arrays (Twenty 24GB SATA SLC flash modules 24GBSTSF-20FM x 32) at a suggested price of $87,521 x 8 = $700,168. This is a major part of the storage array and is an enormous accelerator of IO performance and database throughput. The use of this flash from a benchmark is to be strongly applauded as an improvement in system design. It is the way of the future for database and IO intensive workloads.

Wikibon believes that the IO configuration is a major contributor to the performance and would contribute far more than 17% to the Oracle benchmark figures. From a server performance perspective, the previous conclusion from the TCPH analysis that the two servers are approximately equal is confirmed, with some probability that the IBM system is a little faster.

SPECjEnterprise2010 Price/Performance

As discussed earlier, there is no formal Price/Performance metric for SPECjEnterprise2010, unlike the TPC benchmark where there are stringent guidelines on price determination. The price/performance calculations bring a new level to the Oracle marketing art-form. The Oracle methodology:

• Takes only the Oracle application processors with very little storage to include in the cost calculations;
• Compares that with cost data of IBM processors from another benchmark type (TPC) that was submitted in December 2010, without adjusting for the 3.9% reduction in price over the period between December 2010 and September 2011;
• Ignores the fact that there was significant storage included in the IBM application servers;
• Chooses to ignore the costs of the Oracle/IBM database servers, the Oracle storage servers, and the Sun 5100 flash arrays;
• Chooses to ignore all application software costs;
• Calculates the cost of the Oracle solution as $11.67/EjOPS;
• Calculates that the cost of the IBM solution as $77.97/EjOPS;
• Argues that the Oracle solution is more than six times more cost effective;
• Does not disclose that $/EjOPS is not an official SPECjEnterprise2010 metric;
• Competes with Mike Lynch’s lapses.

Wikibon is unable to work out a true comparative cost of the Oracle or IBM benchmarks from the data available on the Web. However, it is likely that the costs from a server viewpoint are about equal, with some probability that IBM is lower cost.

Bottom Line
The conclusions from the TCPH analysis still hold, and Wikibon suggests that the best conclusion from the data available is that the IBM and SPARC systems are about equal in price/performance for the TCPH and SPECjEnterprise2010 workload measured. There is no support in any way Larry Ellison’s contention that the SPARC RISC server is 1/3 or 1/4 the price of the IBM RISC server.

The World Records of the Seven Sisters

The seven sisters were the following internal benchmarks:-

• Oracle PeopleSoft HRMS Self-Service 9.1,
• Oracle PeopleSoft Enterprise Payroll (N.A.) 9.1,
• The Oracle E-Business Suite R12.1.2 XL,
• The Oracle E-Business Suite R12.1.2 XL,
• Oracle Siebel Loyalty Management,
• Oracle Communications Applications,
• Cryptography.

The full details from Oracle are in the footnotes. The benchmarks and metrics are mainly internal to Oracle, and the increase from one generation to another. Kids the world over get to improve their own world records for the number of swimming pool widths accomplished. Readers are invited to "Find Waldo" from the Oracle information in the footnotes.

Action Item: Senior IT Executives and performance professionals should disregard the Oracle benchmarks and performance claims for the new SPARC systems, and any subsequent comparison. The Oracle T4-4 server is probably performance competitive with IBM, which is a great improvement over the T3 servers. Oracle will have to follow through on their claim of T5 in a year. Oracle SPARC customers with should be pleased with the upgrade path the new servers provide. Oracle customers have the option to delay strategic decisions to convert off the Oracle SPARC platform. Wikibon would not recommend that migration to Oracle SPARC-based systems to installations with high-performance RISC UNIX system from other vendors.

Footnotes: Oracle Information on Seven Performance World Records
Oracle PeopleSoft HRMS Self-Service 9.1:
Oracle's SPARC T4-4 servers posted a world record result of 15,000 concurrent users on this complex OLTP benchmark that emulates a large enterprise with 600,000 employees where HR personnel perform typical HR transactions. The combination of three SPARC T4-based servers running Oracle Solaris, Unicode Oracle Database 11g Release 2 and the PeopleSoft HRMS 9.1 benchmark supported 3.75 times more online users with two times faster response times compared to the best published result from IBM on the previous version, 8.9, of this benchmark (3). The SPARC T4-4 server at the application tier used Oracle Solaris Containers to host two PeopleSoft application server instances and had an average CPU utilization of less than 50 percent, demonstrating a flexible and scalable virtualization environment that has significant headroom for application growth.

Oracle PeopleSoft Enterprise Payroll (N.A.) 9.1:
The SPARC T4-4 server delivered a new world record result on the extra-large volume model of this benchmark using a Unicode Oracle Database 11g Release 2 running on Oracle Solaris. The PeopleSoft Enterprise 9.1 Payroll (North America) benchmark executes large batch workloads typical for a large organization processing employee payroll. The application calculates gross to net earnings, deductions and taxes, and is fully integrated with Oracle's other PeopleSoft Enterprise products. The SPARC T4-4 server with four SPARC T4 3GHz CPUs processed the payroll for 500,000 employees, calculating 750,000 payments in 30.84 minutes, beating the earlier world record result of Oracle's SPARC Enterprise M5000 server by 34 percent with 51 percent better throughput (4).

The Oracle E-Business Suite R12.1.2 XL:
The SPARC T4-2 server and SPARC T4-4 server, deployed in the application and database tier respectively, are the first ever to post results on the on-line Order-Management component of the Oracle E-Business Suite R12 X-large benchmark. Achieving the result of 2,400 users on this business flow demonstrates that SPARC T4-based servers, running Oracle Solaris and coupled with Oracle’s Sun Storage F5100 Flash Array, deliver excellent average transaction response times, are well suited for OLTP-application environments, and leave customers plenty of room for future growth (5).

Oracle JD Edwards EnterpriseOne:
Delivering record-breaking performance on both the interactive and batch components of the “Day in the Life” benchmark that represents the most popular JD Edwards EnterpriseOne applications, two SPARC T4-2 servers running JD Edwards EnterpriseOne 9.0.2 with Tools 8.98.4.2 in Oracle Solaris Containers, and another SPARC T4-2 server running Oracle Database 11g Release 2, delivered a new world record result of 10,000 interactive users, with an average response time of 0.32 seconds, concurrent with 65 batch jobs per minute. This SPARC T4-powered configuration supports 2.5 times more online users and has two times better response time than IBM’s most recently published results with POWER7 CPUs. This solution incorporates a batch processing component that ran six times faster on SPARC T4 servers than on the previous generation of systems, rewarding customers with high performance, scalability and built-in virtualization and security features (6).

Oracle Siebel Loyalty Management:
A critical feature of Oracle’s Siebel Loyalty Management application, the Loyalty batch engine is used by major enterprise customers across the Retail, Travel and Transportation, and Telecommunications industries to support their key loyalty processes and drive customer experience. Three SPARC T4-2 servers and one SPARC T4-4 server with Oracle Solaris, running three Siebel Server 8.1.1.1 instances and Oracle Database 11g Release 2, respectively, achieved 7.65M transactions per hour (TPH). This testing demonstrates that SPARC T4-based servers deliver double the performance of the previous generation of SPARC servers while leaving 75 percent headroom for expansion of the database tier. Oracle Communications Applications: Oracle tested new SPARC T4 servers with two applications instrumental in the order-to-cash business process of communications service providers from its complete portfolio spanning the entire communications systems landscape. Highlights of these tests include: Compared to the previous generation of SPARC servers, two SPARC T4-4 servers running Oracle Communications Billing and Revenue Management demonstrated a 6x improvement in price/performance with 1/5 the data center space requirements and a 37 percent reduction in latency for real-time transaction processing, critical for prepaid mobile operators (7). A SPARC T4-2 server operating at 50 percent CPU utilization and running a single instance of the Oracle Communications ASAP application, with both the application and database tiers consolidated onto a single machine, easily supported the service activation volumes of roughly 1,800 atomic network activation actions/sec, representative of a typical mobile operator with more than 150 million subscribers. On a per processor basis, SPARC T4 can deliver seven times the performance of Itanium processor (8).

Cryptography:
SPARC T4 processors feature new crypto units that support 16 industry standard ciphers and deliver industry-leading, no added-cost cryptography to security conscious organizations in industries such as telecommunications, healthcare and financial services. With new SPARC T4-based servers and the Oracle Solaris ZFS file system, customers can keep data safe and prevent unauthorized access with up to 3x faster encryption than the latest generation of x86 systems (9). Additionally, single-thread OpenSSL security can run up to four times faster on SPARC T4 than on an IBM POWER7-based system (10).