Software testing by statistical methods:
David L. Banks, William H Dashiell, Leonard J Gallagher, Charles R. Hagwood, Raghu N. Kacker, Lynne S. Rosenthal · 1998
Software conformance testing is the process of determining the correctness of an implementation built to the requirements of a functional specification.Exhaustive conformance testing of software is not practical because variable input values and variable sequencing of inputs result in too many possible combinations to test.This paper addresses alternatives for exhaustive testing based on statistical methods, including experimental designs of tests, statistical selection of representative test cases, quantification of test results, and provision of statistical levels of confidence or probability that a program implements its functional specification correctly.The goal of this work is to ensure software quality and to develop methods for software conformance testing based on known statistical techniques, including multivariable analysis, design of experiments, coverage designs, usage models, and optimization techniques, as well as to provide quantitative measures of quality, reliability, and conformance to specifications, including statistical measures and confidence levels.We place initial priority on statistical methods potentially applicable to software defined using conventional techniques, i.e. a functional specification with semantics defined in English, because a large majority of National, International, and Internet standards are specified via non- formal methods, and a major mission of NIST/TTL is to provide conformance testing techniques for such software standards.We focus on conformance testing of software products for which source code is not available; the only access to the product is through a published interface defined by the functional specification, i.e. black-box access.We pursue the potential applicability of several statistical methods to these goals, including binomial models, coverage designs, mutation testing, usage models, and cost-benefit optimization techniques.