Modelling Assistance in Model Driven Software Engineering Tools: A Systematic Mapping Study and a State of the Practice - Raw data and Protocol
David Mosquera, Marcela Ruiz, Óscar Pastor, Jürgen Spielberger · Zenodo (CERN European Organization for Nuclear Research) · 2023
This repository contains the protocol and raw data from the paper "Modelling Assistance in Model Driven Software Engineering Tools: A Systematic Mapping Study and a State of the Practice." This paper is currently under review in the Information and Software Technology Journal. How to use: Inside this repository, you will find five (5) files: DataExtraction.xlsx; MENTI-RESULTS-1.png; MENTI-RESULTS-2.png; StudySelection.xlsx; and Tool-Market-review.xlsx. We describe each file in the following paragraphs. DataExtraction.xlsx. MS Excel file containing the 44 proposals (a.k.a. primary studies) included as a result of our systematic mapping study. This file has one tab named "PrimaryStudies", comprising the following columns: Code: Contains the unique code we use for each proposal. We generate such code following the following template: First and second author last name + two first last name letters of the remaining authors (if present) + year. Ref: Contains the reference number for each proposal used in the paper. Title: Contains the proposal's original title. Status: Always "included" RQ1: Kind of approach. Contains the clusters to answer RQ1: Which strategies do existing proposals follow to assist modelling in MDSE tools? RQ1: Specific Word. Contains the specific words authors use for identifiying their proposal. RQ1: Other? Wich? Contains additional keywords authors use for identifiying their proposal. RQ2: Cluster Limitation. Contains the limitation clusters for answering RQ2: What goals (G) and limitations (L) do the proposals have? RQ2: Limitations (keywords). Contains the keywords authors use to describe their proposals' limitations. RQ2: Limitation(s). Contains full descriptions where authors described their proposals' limitations. RQ2. Goal Cluster. Contains the goal clusters for answering RQ2: What goals (G) and limitations (L) do the proposals have? RQ2. Goal (keywords). Contains the keywords authors use to describe their proposals' goals. RQ2. Goal(s). Contains full descriptions where authors described their proposals' goals. RQ3. Empirically evaluated? Answers yes or no depending on if the proposal was evaluated. RQ3. Evaluation metric cluster. Contains the evaluation metric cluster for answering RQ3: Which evaluation metrics (M) and target users (U) do the proposals have? RQ3. Evaluated metric keywords. Contains the keywords authors use to describe their proposals' evaluation metrics. RQ3. Evaluated metrics. Contains additional information on evaluated metrics for each proposal. RQ3. They define a user? Answers yes or no depending on if the proposal has a target user. RQ3. Cluster by target user. Contains the target user clusters for answering RQ3: Which evaluation metrics (M) and target users (U) do the proposals have? R3. Target user keywords. Contains the keywords authors use to describe their proposals' target users. QA: is the proposal clear? Subjective quality assessment question where: 1 = Yes; 0 = Partially; -1 = No QA: are the limitation(s) clear? Subjective quality assessment question where: 1 = Yes; 0 = Partially; -1 = No QA: Are the goal(s) clear? Subjective quality assessment question where: 1 = Yes; 0 = Partially; -1 = No QA: Are the publications tools downloadable? Subjective quality assessment question where: 1 = Yes; 0 = Partially; -1 = No QA: Is there a clear case study or example illustrating the proposal? Subjective quality assessment question where: 1 = Yes; 0 = Partially; -1 = No QA: Is the whole proposal empirically validated? Subjective quality assessment question where: 1 = Yes; 0 = Partially; -1 = No QA: Are the users clearly described? Subjective quality assessment question where: 1 = Yes; 0 = Partially; -1 = No QA: Are the results clearly explained? Subjective quality assessment question where: 1 = Yes; 0 = Partially; -1 = No CORE/JCR: Objective quality assessment question containing CORE and JCR indexes. Citations: Number of citations. Quality index: Results of calculating the quality index based on the Quality Assessment table (See Table 1 in the paper) MENTI-RESULTS-1.png Contains the results of consulting a set of proposed research questions with the potential target audience of our paper. By asking this question, we aim to answer which RQ target audience found more critical and exciting for dividing the MRQ? This file contains specifically the question: "Which question do you find more important and interesting to solve via a systematic mapping?" MENTI-RESULTS-2.png Contains the results of consulting, which other research questions could be solved via a systematic mapping study. By asking this question, we aim to answer what other RQ would target audience propose to divide the MRQ? This file contains specifically the question: "What other questions would you like to answer via a systematic mapping?" StudySelection.xlsx. MS Excel file containing 2613 records involved in title, abstract, and full text review for selecting primary studies. This file has two tabs: Database-search-data. Contains the results of the screening title, abstract, and full text using the inclusion and exclusion criteria defined in our paper. Moreover, contains the traceability of each decision made for each record and who took that decision (i.e., reviewer). This tab contains the following columns: Year. Contains the year of publication of each record (only included). Strategy. Contains the snowballing strategy if it is forward or backward snowballing. If it is empty, the record comes from the database search. Source code. Contains the study from it comes during the snowballing. First reviewer. Contains the name of the first reviewer. Second review. Contains the name of the second reviewer. Code. Contains the unique code we use for each proposal. We generate such code following the following template: First and second author last name + two first last name letters of the remaining authors (if present) + year. Title. Contains the proposal's original title. Status. Contains the proposals status: included, excluded-after-tr, excluded, and duplicated Include after title screening? Contains Yes if it is included and No if it is excluded. Include after abstract screening? Contains Yes if it is included and No if it is excluded. Include after full-text review? Contains Yes if it is included and No if it is excluded. Include after the second reviewer's opinion? Contains Yes if it is included and No if it is excluded. Include after discussion? Contains Yes if it is included and No if it is excluded. Not fulfilled inclusion/exclusion criteria. Contains the not fulfilled inclusion or fulfilled exclusion criteria as follows: (I1) Does the paper define a specific proposal for assisting users during modelling in MDSE tools? Compilation of proposals like literature reviews, systematic mappings, or systematic literature reviews does not fulfil I1. (I2) Is the proposal designed to assist users during modelling in MDSE tools? We focus on proposals that assist users during modelling in MDSE tools, including—but not limited to—modelling, model tracing, model debugging, model repair, and model validation, among others. Proposals focused on assisting in developing MDSE tools do not fulfil I2. (E1) The proposal’s main contribution is not on assisting users during modelling in MDSE tools. If assisting users during modelling is not the main contribution, we exclude the proposal using E1—e.g., a proposal showing a new MDSE tool where they superficially mention assisting users during modelling. (E2) The proposal is not related to software engineering. (E3) The proposal is not written in English (E4) The proposal is not a peer-reviewed publication (E5) The proposal’s full text is not available. CK. This tab contains the K statistic calculation based on the first and second review agreements. Tool-Market-Review.xlsx. MS File containing the 17 MDSE tools from the Gartner Magic Quadrant reviewed for answering RQ4. What is the state of the practice of modelling assistance? This file contains two tabs: Data. Contains the extracted data for answering RQ4. This tab contains the following columns: MDSE tool. Contains the name of the tool we extract the data from. Moreover, include the GMQ classification inside parenthesis as follows: LE: Leaders; V: Visionaries; C: Challengers; NP: Niche Players. Kewywords. Contains the name MDSE authors refers to their proposals in their tool documentation. Link. Contains the link to the proposal documentation. Quotes. Contains textual quotes extracted from tools documentation. GQM. Contains a picture of the Gartner Magic Quadrant for Enterprise Low-Code Application Platforms 2023. If you have any doubts, contact [email protected] / [email protected]