Supplementary material, Runtime performance data, and Reproducibility script
Anonymous · Zenodo (CERN European Organization for Nuclear Research) · 2024
"participantRuntimePerformanceData.csv" The runtime performance data for participants' solutions as well as the model solutions the authors came up with to problems A and B. partID - the participant ID; any ID prefixed with "fast" is a model solution for A (L1 - L3) or B (L1) problem - the problem (either A or B) mode - whether Copilot was used or not; NULL for model solutions (prefixed with "fast") compiled - whether the code compiled or not (TRUE or FALSE) passed - whether the code passed the sanity checks (TRUE or FALSE or NULL if the code did not compile or ran with errors) IterationNumber - the iteration number for that run. Each participant's solution was run 32 times, so we provided the runtime performance for each run. Runtime (µs) - the runtime performance of the solution for that iteration in microseconds (µs) "studentsProfessionalsSplit.csv" This CSV file contains the compact version of the above data but highlights students vs professionals for summary statistics of only valid solutions (compiled and passed) "supplementary-material.pdf" This file contains supplementary material regarding the associated paper, including, but not limited to, the problems given to the participants, some participants' solutions as mentioned in the paper, the survey questions, and the tutorial before the experiment. "reproducibility-script.zip" This zip file contains a Python script, sample CPP files with solutions, and a README to extract the runtimes as above. Using that script, you can create your own solutions to the problems and run that script against your solutions, to generate the runtimes. It also contains the sample problems as zip files as they are sent to each participant.