The second static analysis tool exposition (SATE) 2009
Vadim Okun, Aurélien Delaitre, Paul E. Black · 2010
The NIST Software Assurance Metrics And Tool Evaluation (SAMATE) project conducted the second Static Analysis Tool Exposition (SATE) in 2009 to advance research in static analysis tools that find security defects in source code.The main goals of SATE were to enable empirical research based on large test sets, encourage improvements to tools, and promote broader and more rapid adoption of tools by objectively demonstrating their use on production software.Briefly, participating tool makers ran their tool on a set of programs.Researchers led by NIST performed a partial analysis of tool reports.The results and experiences were reported at the SATE 2009 Workshop in Arlington, VA, in November, 2009.The tool reports and analysis were made publicly available in 2010.This paper describes the SATE procedure and provides our observations based on the data collected.We improved the procedure based on lessons learned from the SATE 2008 experience.The changes included random selection of subsets of tool warnings for analysis and also selection based on human analysis, more detailed analysis categories and criteria, an enhanced output format that provides a richer description of weakness paths, and a more detailed and accurate analysis of tool warnings.The SATE data suggests that while tools often look for different types of weaknesses and the number of warnings varies widely by tool, there is a significant degree of agreement among tools for well-known weakness categories, such as buffer errors.The data also provides evidence that, while human analysis is best suited for identifying some types of weaknesses, tools find a significant portion of weaknesses considered important by human experts.This paper identifies several ways in which the released data and analysis are useful.First, the output from running many tools on production software can be used for empirical research.Second, the analysis of tool reports indicates actual weaknesses that exist in the software and that are reported by the tools.Finally, the analysis may also be used as a basis for a further study of the security weaknesses and of static analysis.