Automated Item Generation by Combining the Non-template and Template-based Approaches to Generate Reading Inference Test Items
Eunjin Shin · ERA: Education and Research Archive (University of Alberta) · 2021
Automatic item generation (AIG) is an area of research, where cognitive and psychometric modeling practices are used to create test items with the aid of computer technology. AIG can produce a large number of test items to support the surging demand for test administration. Two general methods are available for producing items using automated processes. The methods vary in their use of templates to structure the content. While the two frameworks could provide significant paradigm shifts to generate test items, the type of test items and the applicability of the items to operational administration are limited. To overcome such limitations, a hybrid AIG framework was created that extends the capacity of template-based AIG with rich natural language processing analyses introduced in the non-template-based AIG systems. The new framework is applied to produce test items in reading comprehension item generation, which is considered a complex and challenging task for previous AIG systems. More specifically, the current method disambiguates an underlying subtopic structure from narrative stories—the Harry Potter series—using topic modelling analysis, a weighted Latent Dirichlet Allocation approach. Then, the disambiguated subtopic information is logically combined and arranged using item models from template-based approaches to generate reading inference-type items. This study has the potential to contribute to the methodology and the current practices of automated item generation by highlighting the importance of integrating two primary components–item models and natural language processing techniques–to generate test items in the previously challenging domain of reading comprehension.