Open Benchmarks?

Even though MySQL is an open source company, there are some areas where we could do a better job being even more open. One of these is more regular publication of database benchmarks. Oddly enough, it's an area where MySQL shines.

When I joined MySQL in 2003, one of the reasons that attracted me was that even without a lot of marketing, it was clear what the major advantages of our products were: performance, reliability and ease of use. And of course, MySQL is open source. Those product characteristics were reinforced whenever I spoke to customers, partners and employees about MySQL. There were also some good outside proofpoints such as a benchmark by eWeek that provided credible third-party validation. We weren't just drinking our own Kool-Aid, so to speak.

But that was a while ago, and the eWeek benchmarks, while great, were getting bit old. We'd released MySQL 4.1 which had a lot of new capabilities, we'd done some ongoing performance tuning and so we needed to do some updates to the benchmarks.

At MySQL we have a small benchmarking team, which has some of our best performance tuning engineers. They do amazing things for customers. They can show up for a couple of days and double the performance of huge, complex systems. I don't know how they do it, maybe it's some special Russian brew they drink, but the customers are always blown away. And they routinely help companies like Intel, Sun, HP improve the accuracy of the benchmarks they run with MySQL.

Lies, Damn Lies and Benchmarks

But the downside of having such a technically skilled team is that they don't want to publish benchmarks unless they are perfect. There's a common view among engineers that everyone knows benchmarks are not realistic. (Ok, what they actually say is "Benchmarks are BS".) Some tests, like TPC, are more a measure of your hardware budget than of database performance. So in many engineers' minds, instead of running theoretical benchmarks we should just tell people to run their own tests that more accurately reflect how they intend to use MySQL. (Of course, this could also be solved if we just added "two more guys", the common refrain on any project suggested at MySQL.)

In the ideal world, everyone would run their own benchmarks and the results would always be completely representative of the application scenario under consideration. But in the real world, customers and prospects continue to ask us for benchmarks. They want to have some base level for assessing performance and comparing scalability on different platforms or architectures. While no benchmark is perfect, most of our customers are savvy enough to take the results with a grain of salt. But as one customer put it, "We want a starting point to know how we're doing with our own tests."

So a few months back we published a white paper that gathers some of the various benchmark results, tests done by Sun, SGI, BEA, eWeek, and others and put that on our web site. It includes links to the various tests and results. Since engineers do not trust benchmarks, you can download the source code and benchmark your own system to your heart's content. If you do, I hope you'll blog about the results, whether they are good or bad.

Will our benchmarks reveal areas where our code isn't quite perfect? Sure. But the good news is, then we can fix it. MySQL has always benefited from openness, whether it's finding bugs or fixing "gotchas" that don't quite work as expected. And I think benchmarks fall into that same area. We may as well be open about it because then we can work together to improve things.

An Open Benchmark Challenge

Although we're open with our benchmark results, that's not the traditional policy in the database industry. Traditionally, closed source database licenses including those of Oracle, Microsoft and DB2 prohibit the publication of benchmarks without permission. (This is known as the "Dewitt clause" named after the author of one of the first major public benchmark papers published at the 1983 ACM Conference on Very Large Databases (VLDB)). I challenge Oracle, Microsoft and IBM to give their customers the freedom to publish any benchmarks they choose. After all, shouldn't they trust their customers enough to know how to properly run these tests?

In the coming months, we'll put publish more benchmark results that demonstrate the performance of MySQL 5.0 under a variety of different configurations and application scenarios. But benchmarks are like statistics; they give you a slice of data and its up to you to draw the right conclusions. And keep in mind what business professor Aaron Levenstein once said of statistics: "Statistics are like a bikini. What they reveal is suggestive, but what they conceal is vital."

And please let me know your thoughts on what benchmarks you think MySQL should publish. What do you do in the real world for benchmarks?