Fundamental: Optimizing a Teacher Professional Development Program for Teaching STEM with Robotics Through Design-based Research
S. M. Mizanoor Rahman, Veena Jayasree Krishnan, Vikram Kapila · 2020
Abstract This paper illustrates the use of periodic program evaluation and feedback to measure and optimize the design and implementation performance of a professional development (PD) program for in-service teachers on teaching STEM with robotics through design-based research (DBR). We recruited 23 in-service middle school science and math teachers and designed a three-week, eight-hrs. per day, PD program for participants to learn how to develop robotics-aided lessons and how to implement the developed lessons in a classroom environment. Under the iterative refinement framework of DBR, we divided the three-week PD program into three successive iterations, i.e., each week as an iteration. We selected to express the quality (performance level) of the design and implementation of the program using several key evaluation criteria (key performance indicators or KPIs), e.g., quality of instruction by the instructors, quality of instruction materials, selection of appropriate mode/method of instructions, scheduling and time management, addressing equity, diversity and individual needs, balance in delivery of contents (appropriate combination of math, science, education, and robotics topics), maintaining pre-requisites, quality of reflections and addressing feedbacks from participants, quality of surveys and assessments, team building and teaming, proper guideline toward innovation in new lesson planning, and level of engagement of the teachers. We developed an objective function based on the above criteria and assigned weight to each criterion. We used two separate Likert scales with scores between 1 and 10 to determine the weights and assess the performance level of each criterion through teacher-surveys. We conducted the surveys at the end of each week (iteration), determined the objective function value (OFV), analyzed the outcomes, and took necessary actions to enhance the OFV in the next iteration(s). The maximum possible OFV (OFVmax) for the selected KPIs, weights, and scales is 1,200. We assume that the OFVmax is the optimal or targeted performance level, and out of the 3 weeks, the week with the highest OFV shows the achieved local optimum performance level. We assume that the constraints are the same/similar in each week. We adopt the following two hypotheses. H1: The DBR process causes the OFVs to increase (i.e., with successive refinements, the teachers perceive the program to improve) toward the optimal value as the iteration propagates. H2: The weights of the performance criteria as perceived and evaluated by the teachers vary between iterations. The results show that, guided by the average scores of weights and scores assigned by teachers to KPIs, the DBR refinements yield the OFVs for the first, second, and third iterations as 803, 859, and 897, respectively, thus validating the first hypothesis. To examine the second hypothesis, variations in weights perceived by the teachers between the iterations are analyzed using paired t-tests. Variations in perceived weights in each iteration are also analyzed. The proposed approach is novel in that it acts as an assessment tool that may bring objectivity in the design and implementation of PD programs. The DBR can act as a continuous improvement tool to enhance the quality of the PD program iteratively. Both approaches can jointly enhance the effectiveness and performance of the PD program. The proposed integrated (quantitative and qualitative evaluations and DBR) approach can be used to enhance the effectiveness of other PD and educational programs and courses.