Confidence Intervals for Subjective Tests and Objective Metrics That Assess Image, Video, Speech, or Audiovisual Quality
Margaret Pinson · 2020
This report describes a methodology that measures the precision of objective metrics that assess image quality, video quality, speech quality, or the overall audiovisual quality. We assess the confidence intervals of 60 subjective tests and use a confusion matrix to classify the conclusions reached when two subjective test labs perform the same experiment. This allows us to compute the metric’s confidence interval and, when confidence intervals are used to make decisions, to prove whether the metric performs similarly to a subjective test with 15 or 24 subjects. When confidence intervals are not used, the metric’s precision is likened to a certain number of people in an ad-hoc quality assessment. The methods in this report are developed and evaluated using speech quality, video quality, image quality, and audiovisual quality datasets.