Benchmarking the Cloud: A Users Perspective
Khushal Dave · TRAP@NCI (National College of Ireland) · 2014
The increasing popularity of cloud computing and various options have manifolded over time. The foremost question in front of cloud users is how to precisely measure the performance capabilities of various options. The criteria for measuring performance could vary from measuring cost to floating point operations per second (FLOPS) or data manipulation (I/O). Present benchmarking tools like Linpack, which measures performance in FLOPS cannot evaluate the performance of multi-core instances. Being non-adaptive, these bench- mark do not take system configuration and environment into consideration during testing. Currently, there are no properly designed benchmark suite which suitably adapts itself according to the node resources. This research work focus on the designing of a scientific benchmark suite which will provide the maximum FLOPS performance of a CC over various CPU intensive tests. This will allow us to evaluate the performance on criterias like FLOPS and effect of multi-threading. This approach is completely automatized thus can be used on any compute instance starting from single node to multi-node instances. The intelligent identification of suitable load data for individual tests which is automatically decided by the resource specification and the execution environment. Here we are using different data types to simulate CPU intensive tasks. This benchmark suite can be suitably used for any instance size without modification. The results obtained from these tests will allow the user to identify the suitable option amongst various CC. The tool developed has been tested on Openstack and Azure instances and was able to provide the correct results to compare the performance. This can be easily used by any naive user to compare the performance of various CC's thus, allows easy comparison of performance.