The 1st Workshop on Testing Technologies and Tools for Critical Industry Applications Improving Test Quality by a Test Type Analysis Based Method
Qin Liu, Wenqiang Zheng, JunFei Ma · 2009
This paper describes an easy adapted method of improving test coverage and effectiveness in an on-going indus- trial project by introducing and applying 'test type' into test case design and execution. It presents comparison results of the number of system test item, effectiveness of test cases, and defect detection percentage between two increments in a real-world telecom project. It argues the importance of carrying out a test type analysis approach for test design, and proved a better test effectiveness (improved 5%) and DDP(improved 14%) can be performed by implementing the proposed approach. There was only a slightly increase of the budget after adapting the approach. Index Terms—Test analysis method; Test Type; Test effective- ness; DDP; The paper introduces a study based on an industrial testing project of a telecommunication system (IPDSLAM), which provides BroadBand (ADSL) Access service based on pure Ethernet Switch for customers. The system provides rich protocols and features in layer 2 and 3, and its overall development is based on an incremental model (1). There are four increments which contains a large number of new features in each increment. After the first increment of the product, feedback from clients was not positive. The clients complain about the large number of post-release defects, which might damage the organization's well-established image. In a project meeting, the question why those post-release defects were not caught during the test phase? was bound to arise. Two experienced testers were assigned to identify the roots of deficient in test coverage (2) and effectiveness, and improve them in the upcoming increments. After observing and analyzing executed test cases and results, a test type based test case analysis and design method was recommended and had being applied in the third increment of the project. Section 2 presents the observation and analysis of historical increment; section 3 proposes a test type based analysis method; section 4 illustrates comparative test results between the increments with and without proposed method; and section 5 draws a conclusion in an improvement of effectiveness by applying the proposed method.