Data-driven Program Analysis Deployment
Anton Ljungberg, David Åkerman · Lund University Publications Student Papers (Lund University) · 2020
Program analysis is useful for reporting code defects that can be hard or time consuming for a developer to nd, but usability issues make many developers choose not to analyze their code with such tools.False positives introduce a lack of trust in reported defects.When users can not trust reported defects, they need to spend time on defect validation.Incomprehensible and excessively large results gets in the way of the development process.A promising approach addressing the usability issues of program analysis tools is to adapt the tools to the needs of users by making data-driven improvements.In this thesis we have created, deployed and evaluated a data-driven program analysis system.We have implemented a system named MEAN (Meta Analysis), together with a handful of protocols for running standardized program analysis within a variety of tool stacks.MEAN has been deployed at Axis Communications with code review as an integration point, where program analysis alerts reached the developers.Analyzers were continuously con gured in response to user feedback on analyzer results.Many alerts addressed defects which were not introduced by the change where they were presented.With this stated, users of MEAN xed one out of ten defects and actively reported alerts as not useful once per alerts.When isolating defects introduced by the current change, one out of three defects were xed and one out of alerts about these defects was reported as not useful.Fixing a defect introduced by an older change would often make the current change incoherent, while reporting an alert about such as not useful did not have this negative implication.The evaluation of the deployment verify that noise in analyzer results, including false positives, can be reduced by making data-driven improvements.The evaluation also underline that users of program analysis integrated into daily development shall not be presented with program analysis alerts that are unactionable and redundant.