With Andy Pavlo, Djellel Difallah, and Phil Cudre-Maroux we just completed an interesting project: an extensible testbed for benchmarking relational databases.
This was motivated by the huge amount of time each one of us wasted in our research in building workloads (gathering real/realistic data and query loads, implementing driving infrastructures, collect statistics, plot statistics).
To reduce the wasted effort for others and to promote repeatability and ease of comparison of scientific results we pull together our independent efforts and created an extensible driving infrastructure for OLTP/Web workload, capable of:
- Targeting all major relational DBMSs via JDBC (tested on MySQL, Postgres, Oracle, SQLServer, DB2, HSQLDB)
- Precise rate control (allows to define and change over time the rate at which requests are submitted)
- Precise transactional mixture control (allow to define and change over time % of each transaction type)
- Access Distribution control (allows to emulate evolving hot-spots, temporal skew, etc..)
- Support trace-based execution (ideal to handle real data)
- Extensible design
- Elegant management of SQL Dialect translations (to target various DBMSs)
- Store-Procedure friendly architecture
- Include 10 workloads
- AuctionMark
- Epinions
- JPAB
- Resource Stresser
- SEATS
- TATP
- TPCC
- Wikipedia
- YCSB
If you are interested in using our testbed, or to contribute to it visit: http://oltpbenchmark.com/