Comparative benchmarks: here there be dragons Comments Off on Comparative benchmarks: here there be dragons

Part of the work I’m doing for a client at the moment involves a migration of databases from a Sun E2900 running Solaris 8 to a cluster of M3000s running the latest build of Solaris 10. This is going to be a long post, but I want to touch on some of the difficulties businesses face when trying to benchmark such migrations.

Initially the migration was sized on the simple (and flawed) premise that we have 24 cores running at 1.9Ghz (US IV+), and we’re now moving to 8 cores running at 2.9Ghz (SPARC64 VII). Therefore the box will be X times faster, where X is a nice big number that’s acceptable to the business. Project signed off, cheque printed – next!

This is great, but from a technical point of view it’s dangerously misleading – and it also leads to some pain when trying to manage this infrastructure. Could we consolidate more databases onto this machine? What’s the performance implication? What’s the real performance gain we can expect from the migration, and what are the key factors that determine this?

First of all we have the OS. Solaris 8 is the Honda Civic of operating systems. It’s been around for ages, it’s reliable, it’s not fast and flashy, and you can try pimping it out but you’ll end up wasting money and looking foolish.

Solaris 10 has so many improvements in areas that boost performance (as well as decreases the administration pain) that they might as well be different OSs. The only way to do a proper comparison is to sit down with the same hardware, and the same application, and run through some performance tests with both 8 and 10, then compare the results.

The problems for all businesses is that this just isn’t possible – this is a stable production machine. No-one is going to buy another one when they’re going to migrate to new tin, and when the machine becomes freed up it will be post-migration – which would be too late.

Then you’ve got the actual hardware. The UltraSPARC IV was a cracking bit of kit, but it’s been out for almost a decade now, and Fujitsu’s SPARC64 is faster in every area. There’s a huge technology gap, and again, unless we get the chance to compare the same OS with the same application on the two different sets of CPUs – but then the server hardware is so radically different that the results will be out of whack.

Let’s add into the mix some performance improvements within the M3000 itself to highlight this. Initially the SPARC64 VII CPU that was offered has a clockspeed of 2.52Ghz, but the kit for this particular project has 2.75Ghz CPUs. That’s only a 9% performance increase, right?

*bzzzrt* Computer says “No”:

• faster System clock at 306MHz
• faster memory – now using 667 MHz DIMMS (the memory runs at 612 Mhz, doubling the system clock)
• faster interconnect to the Jupiter System Controller at 1224MHz (four times the system clock)

This actually adds up to a 23% speed increase for the CPU overall. This highlights the effectiveness of balanced RISC computing platforms, compared to the single minded “clock speed is king” focus of x86. It doesn’t half make of a mess of your benchmarking figures, though.

Mr. Benchmark has a great post that goes into far more depth about this sort of thing, and I highly recommend reading his blog.

The conclusion? Simple “compare speed and number of cores” comparisons are very, very dangerous, and can lead to a massive over-spec of new systems. However, many times this will be the only way to make such a comparison, unless you can get early access to a vendor’s test lab. In short, tread carefully, and think carefully about *all* aspects of system performance.