Forecasting System Capability Using the Scenario-Based Experimental-Design Test Approach
Ya Lun Li, Blake Roberts, Michael W. Grenn · IEEE Systems Journal · 2018
During the requirement-based test approach (RBT &E), it is difficult to objectively and quantitatively forecast system-of-interest (SOI) capabilities (or ability to achieve system requirements under its stated operational environment) through interpretations and/or analysis of obtained element verification data. This challenge remains because RBT&E verification data are collected under unique verification conditions and measurements specific to only the allocated element requirements (e.g., element A's demonstrated y1-mi/h speed under x1°C verification condition versus element B's demonstrated payload of y2-tons at x2-m altitude). Such mismatched and conditional variables cannot be mathematically amalgamated to quantitatively forecast SOI capability. The proposed scenario-based experimental-design (SBED) test approach provides the ability to objectively and quantitatively forecast SOI capability and obtain matching element verification data (i.e., all normalized and standardized to a common input domain and a common unit). By applying design of experiments test methods and response surface methodology statistical techniques, the SBED test approach can model and output element verification data as “system-element capability” responses (or ability to fulfill element requirements) influenced by SOI operational-environment's parameters. It then forecasts the SOI capability (as depicted through sequencing of all elements' interactions) by a Boolean algebraic sum of all the (now matching) “system-element capability” response models. SBED introduces two systems engineering and test and evaluation (T&E) benefits. First, the early delivery of an objective and quantitative SOI capability forecast that has been previously unobtainable; second, more-descriptive element-capability regression models (with respect to the entire SOI operational environment vice just the requirement-specific condition per RBT&E) to assess and mitigate SOI design risks and uncertainties. To demonstrate these advantages, the authors apply SBED's data analysis techniques to an existing U.S. Air Force flight test's dataset and compare the SBED-produced quantitative SOI capability forecasts against the test's original qualitative conclusions.