Before making a decision to change platforms or software based on published performance benchmarks, consider creating your own testing environment to ensure that the evaluation includes all criteria important to you. Because it is nearly impossible to simulate all the ways an application will be used, understanding the scenarios included in a test is critical to estimating performance in production.
Recently, I investigated an alternative datamart for the SessionM platform. Since optimal system performance is always our goal, I became interested in Amazon’s claim that their Aurora database is five to eight times faster than our current database of choice: MySQL.
Amazon states, “Aurora uses a variety of software and hardware techniques to ensure the database engine is able to fully leverage available computer, memory and networking. I/O operations use distributed systems techniques such as quorums to improve performance consistency. Testing on standard benchmarks such as SysBench have shown up to a fivefold increase in throughput performance over stock MySQL 5.6 on similar hardware.”
Our current performance requirements necessitate heavy use of indexes and query optimization. When we add new tables or users to the datamart, we generate new indexes and evaluate all queries to ensure table access is optimized.
An eightfold performance improvement would bring our unindexed survey time below the one second threshold. This would make it easier to develop and move more queries to the datamart. It would also make it less likely that a change to the front end causes a performance problem. At first glance, it seems obvious that a move to Aurora would be a good thing for us.
I decided to test the two databases in our environment to ensure that the performance increase is great enough to support a move to this new platform.
I set up four systems, each with production-sized data. Since we currently use an r3.xLarge MySQL database for our datamart, I used the following test systems:
I selected the following queries to calculate our environment-specific benchmarks:
The results of my performance test show that Aurora is only about two times faster than MySQL for our indexed queries. In addition, MySQL outperforms Aurora for queries without indexes. This outcome does not fundamentally change the way we work with the datamart and does not warrant a move to a different platform.
| Survey
(No Index) |
Survey
(Index) |
Insights
(No Index) |
Insights
(Index) |
|
| MySQL xLarge | 3.1 | .23 | 3.8 | .5 |
| Aurora xLarge | 6.3 | .14 | 19.4 | .21 |
| MySQL large | 3.8 | .5 | 5.2 | .37 |
| Aurora large | 6.5 | .15 | 20.5 | .35 |
Our benchmark test measures read performance for a relatively large query (1 to 3 queries per second) with no concurrency.
The benchmark performed by Amazon tests, performance for lots of queries (hundreds of queries per second), write performance, and highly concurrent queries.
If our application’s use of the database/datamart was the same or similar to that of the Amazon test application, I expect we would have seen similar results to the benchmark they provided. And it would make sense to move to Aurora to realize a significant performance increase. However, since our system is quite different from the ones in Amazon’s scenario, the performance change does not justify a move from our current database platform.
Amazon’s benchmark did provide accurate data about some performance aspects for a specific audience. However, the results must be taken in context, and all criteria reviewed, when deciding if changing technology is necessary. To really use a benchmark’s results for decision-making, ensure you know the exact assumptions made on the test-bench.
Even the best benchmarks can simply give you an approximation of how well software or hardware does in a specific circumstance. With well-intentioned benchmarks, bias can still appear.
Suggestions