Standards of Probability Sampling for Legal Evidence
William Edwards Deming · The American Statistician · 1958
These STANDARDS have appeared as a special report from The Society of Business Advisory Professions. They had their origin in the need expressed by the lawyers and accountants of that Society for an authoritative statement that would dispose of the question that often comes up on whether data obtained by a sampling survey are admissible as evidence. The answer contained in the STANDARDS places the test of acceptability squarely where it belongs: (1) on the equal complete coverage, the question being whether, had the study been an equal complete coverage, would it have been acceptable as evidence? (2) on the question of whether the sample was a probability sample; (3) on the precision, as measured by the standard errors; (4) on the magnitudes of the nonsampling errors, these being common to both samples and complete coverages. These STANDARDS establish specifications for sampling, which if certified as followed, and provided the standard error of the sample-result is small enough to be innocuous, and provided its interpretation takes account of the number of degrees of freedom if small, and of extreme skewness in the frame, if it exists, should establish the acceptance of the numerical result of a sample as legal evidence on the same status as the result of an equal complete coverage of all the units in the same frame whence the sanmple came, without the requirement of a dissertation on the theory, techniques, meaning, and reliability of sampling. Questions on the usefulness of the data, the date of the survey, the procedure of interviewing or of testing, and the decision on whether a proposed frame covers adequately the universe that one really wishes to study, are the same whether the study will be a complete coverage or a sample, big or little. These questions are not solved by statistical theory, but belong to the realms of subjectmatter, law, and administration. A careful statistician will of course satisfy himself on these points; else the results may lack utility. Standards and codes define responsibility; not ways in which the statistician and the expert in the subject-matter work together. A new concept in the STANDARDS is the equal complete coverage, defined as a coverage (a 100% sample) of all the sampling units in the same frame as was used for the sample, under the stipulation that the complete coverage be carried out with the same procedure as was used for the sample for eliciting the information, with the same definitions and with the same care and over about the same period of time. The sample contains omissions, nonresponse, correct response, and wrong response, in about the same proportions as would be found in the equal complete coverage. The sample comes from the frame. Objective statistical inferences from the sample refer only to the frame; not to the universe, unless the frame covers the whole of the universe. The standard error measures the margin of difference, for a stated probability, between the equal complete coverage and the result of a sample. The outside margin of difference between the result of the equal complete coverage and an estimate made from a sample thereof is 3 standard errors of the estimate. The actual difference is usually within 2 standard errors, and may lie in either direction. One must however take care in this interpretation to make proper correction if the number of degrees of freedom in the estimate of the standard error is small, and must make further correction by the proper theory if the populations of the sampling units in the frame show extreme skewness. A small standard error means that an equal complete coverage of the same frame would give very nearly the same result as the sample. It does not mean that the data will be useful, nor that the field-work was carried out expertly, nor that the responses were accurate, nor that the problem of nonresponse was conquered. Random drawing implies the use of random numbers for the selection of the sampling units from the frame. There exists today no other satisfactory operational definition of random drawings. The STANDARDS make the important point that once the standard error of a result is estimated by a valid procedure, and if it is interpreted with due regard for the number of degrees of freedom, and with further regard for extreme skewness if it exists, the details of the sampling plan add no new information concerning the margin of sampling error. This statement should be of special interest in marketing research, where clients often specify needlessly and often inexpertly the size of the sample and the procedure of sampling, when their real concern should be the standard error or the amount of information desired for some of the results of chief importance. It should also be of interest to marketing research companies themselves, which often present to their clients not only the standard error but the procedure of sampling as well, as if the procedure added something to the standard error. Actually, what is really important, other than the standard error, is an evaluation of the nonsampling errors.