Evaluation of Digital Repositories from an End-users' Perspective: The Case of the reUSE project
Mateja Šmid · International Conference on Electronic Publishing · 2006
This demonstration presents the results of end-user survey carried out in the reUSE project, in which we examined the usability of three different digital repositories from Austria, Estonia and Germany. The reUSE is a cooperative project from eContent scheme and involves national and university libraries and universities from Austria, Estonia, Germany and Slovenia. Evaluation of the digital repositories was one of the main goals. Overall aim of the evaluation was to make user-centred repositories which will be at the same time most efficient in technical and organizational regards. An end-user evaluation was carried out in reUSE project demonstrators' libraries of the three repositories and because obtained data are confidential, they will be marked only as Case 1, Case 2 and Case 3. Case 1 presents digital repository of university library and their specific is that they provide digital material also for blind users. At the moment of evaluation there were about 3000 digital documents in the repository. Their most important users are students, professors, scientists and librarians. Case 2 is the repository of the National library with much broader population in terms of age, profession, and use habits. Since it has only been running for a few months, it is understandable that contained merely 196 digital documents. Case 3 is again running by university library and is the oldest digital repository evaluated within the project. At the time of the end-user evaluation nearly 2800 documents were there (especially thesis and scientific papers). Their end-users are coming from the university environment with very specific needs for scientific material and scholarly publications. Since it was very complicated and expensive to carry out a case study in special test rooms and because the survey had to be done in national languages, we designed a special questionnaire for end-users based on ISO 9241-11:1998 (Usability standard) [3] and ISO 13407:1999 (Human-centred design processes for interactive systems) [4]. The questionnaire consists of a demographic part and of the questionnaire, which is the main focus of the survey. The specific research objectives were to obtain information on content, recognition, performance and efficiency, personal and subjective perception and error tolerance. An intensive test phase was carried out in June 2005 and in the end of September 2005. It was expected that during the summer of 2005 implementers would fill the repository with additional content which should influence the results of the 2nd evaluation period. Paper-based questionnaire were sent to the Case 1 and Case 2 libraries while for Case 3 the same questionnaire was transformed into an on-line form. Librarians of each repository helped to disseminate the questionnaire among their users. In order to get a stable and representative sample we required approximately 100 questionnaires of end-users. However, this goal turned out impossible to reach, especially for Case 3 because it was very difficult to retrieve data from on-line questionnaires. It was very hard to attract end-users to fill in the questionnaire via the digital repository’s interface. Paper based questionnaires and personal contact still provided more efficient feedback. Besides, paper based questionnaires were more completely filled-in than the on-line ones. For the detailed data analysis software Remark Office OMR for statistic researches was used. Overall, end-users were happy to have a digital repository and to use it at their work or in studies, but they argued that there are too few items in the repositories. As expected, end-users in the national library are older and of different ages in comparison to the end-users of university libraries. However, the majority of end-users were born in the 60s and 70s. Further analysis shows that most of the end-users have intermediate experience with computers and on average use the repository only a few times a year. The number of novices is very low.