A Comparative Analysis of Query Generation Methods for IoT Middleware Evaluation
Ravindi de Silva, Arkady Zaslavsky, Seng W. Loke, Prem Prakash Jayaraman · 2024
The rapid expansion of Internet of Things (IoT) applications necessitates the demand for efficient IoT middleware platforms, especially Context Management Platforms (CMPs) for accessing real-time contextual information. Current IoT middleware evaluation methods lack accessibility, posing a significant challenge for standardized performance analysis of CMPs. This paper introduces a novel scenario-based context query generation approach for assessing the data retrieval performance of a CMP. The proposed approach enables scene creation using either IoT data or real-world images, employing both data-driven and image-driven scene generation methods. This paper presents the challenges, limitations, and applicability of each approach in scene generation through a qualitative evaluation. The scenes are transformed into scene graphs, enabling inference to identify situations. Both approaches generate queries reflecting dynamic real-world situations, allowing for random query loads to evaluate CMPs fairly. This paper validates the proposed approach using real-world IoT datasets and camera images with a focus on bicycle riding safety use cases. The evaluation provides insights into the contextual completeness of scene graph representations crucial for accurately capturing real-world situations. The proposed approach generates queries of varying complexity, with the only difference being the underlying mechanism of scenario generation, providing a streamlined and user-friendly method for evaluating the data retrieval performance of CMPs.