Automatic refinement of user requirements : a case study in software tool evaluation
Jacob L. Cybulski · Journal of the Association for Information Systems · 2002
This paper presented a sy stematic approach to evaluating outcomes of the SDRM research project (Software Development Research Method - Nunamaker, Chen, et al. 1991). The empirical method adopted in this evaluation is grounded in the belief that simple sy stem testing is insufficient in a development-based research. Ty pically testing focuses on showing the method or a tool to be correct or efficient, however, the investigation of the proposed method or a tool impact on its target audience is frequently either missing or is conducted in an indirect way . As an alternative to the commonly used approaches, this paper explored and fused a number of different empirical methods to define and develop a testing environment for the proposed method/tool, to calibrate test data so that the test results could be assessed and compared, and finally to test and evaluate the method/tool in the well-designed and calibrated environment. As the RARE IDIOM project focuses on using information retrieval techniques to support requirements refinement into reusable designs, the evaluation process involved definition of suitable problem (for requirements) and solution (for designs) domains, populating this domain with fully described and classified concepts and artefacts, calibrating the quality measures of requirements refinement with respect to the expert and novice design decisions, and finally evaluation of the method/tool by showing its performance to be approaching that of experts and significantly exceeding that of novices. The paper thus demonstrated the effectiveness of the RARE IDIOM method/tool. Even more importantly , it also illustrated that development-based research, as exemplified by the SDRM method, can be grounded in the sound empirical proc...