Usefulness and usability of heuristic walkthroughs for evaluating domain-specific developer tools in industry: Evidence from four field simulations
Olaf Leßenich, Stefan Sobernig · Information and Software Technology · 2023
The usage of domain-specific languages (DSLs) is an approach to reduce complexity in software development by expert developers for selected application domains. To support expert developers, a DSL is often combined with a tailored, domain-specific developer tool, offering similar functionality as general-purpose programming environments (IDEs like Eclipse for Java) while integrating with a domain-specific toolchain (e.g., code or documentation generators, simulators). General-purpose development environments have been successfully evaluated for the programmer experience (PX) and for anomalies using heuristic walkthroughs as a mixed review technique combining cognitive walkthroughs and a heuristic evaluation. In this paper, we report on the usefulness and acceptance of heuristic walkthroughs as an PX evaluation technique applied to domain-specific languages and IDEs in an industry context. Heuristic walkthroughs are used in four interventions (field simulations) to assess the programming experience and usability of domain-specific, Eclipse-based IDEs in a concrete industry setting. Data on the usefulness and acceptance (perceived satisfaction) of the walkthroughs themselves are collected and analysed. Our studies show that, in practice, the instrument of walkthroughs is useful for revealing practically relevant PX anomalies, while maintaining acceptance by practitioners participating in the walkthroughs. The documented variant of heuristic walkthroughs is eligible to become adopted for future field studies in academic research and for evaluation projects in industry, in support of developing domain-specific developer tooling in an evidence-driven manner.