Benchmarking Database Systems A Systematic Approach
Dina Bitton, David J. DeWitt, Carolyn Turbyfill · 1983
This paper describes a customized database and a comprehensive set of queries that can be used for sys-tematic benchmarking of relational database systems. Designing this database and a set of carefully tuned bench-marks represents a first attempt in developing a scientific methodology for performance evaluation of database management systems. We have used this database to perform a comparative evaluation of the database machine DIRECT, the "university " and "commercial " versions of the INGRES database system, the relational database sys-tem ORACLE, and the IDM 500 database machine. We present a subset of our measurements (for the single user case only), that constitute a preliminary performance evaluation of these systems. NOTE TO THE READER It is important for the reader to recognize that the results presented in this paper represent the performance of the various database systems at ONE point in time and that new releases of the various systems will undoubtably perform differently. The objective of this research was not to make a definitive statement as to which is the best relational database system on the market today. Rather, our goal was to develop a standard set of benchmarks that could be used by database system designers for evaluating changes to their systems and by users for selecting the system which best suits their needs. It is also imperative that the reader understands that the results presented in no way measure the performance of the various systems in a multiuser environment. We are currently developing a methodology for benchmarking database systems in this environment.