Evaluating test suite characteristics, cost, and effectiveness of FSM-based testing methods

Author(s)

Endo, Andre Takeshi ^[1] ; Simao, Adenilso ^[1]

Affiliation

^[1] Univ Sao Paulo, Inst Ciencias Matemat & Comp, BR-13560970 Sao Carlos, SP - Brazil

• Document type: Journal article
• Source: INFORMATION AND SOFTWARE TECHNOLOGY; v. 55, n. 6, p. 1045-1062, JUN 2013.
• Web of Science Citations: 12

Abstract

Context: Testing from finite state machines has been investigated due to its well-founded and sound theory as well as its practical application. There has been a recurrent interest in developing methods capable of generating test suites that detect all faults in a given fault domain. However, the proposal of new methods motivates the comparison with traditional methods. Objective: We compare the methods that generate complete test suites from finite states machines. The test suites produced by the W, HSI, H, SPY, and P methods are analyzed in different configurations. Method: Complete and partial machines were randomly generated varying numbers of states, inputs, outputs, and transitions. These different configurations were used to compare test suite characteristics (number of resets, test case length) and the test suite length (i.e., the sum of the length of its test cases). The fault detection ratio was evaluated using mutation testing to produce faulty implementations with an extra state. Results: On average, the recent methods (H, SPY, and P) produced longer test cases but smaller test suites than the traditional methods (W, HSI). The recent methods generated test suites of similar length, though P produced slightly smaller test suites. The SPY and P methods had the highest fault detection ratios and HSI had the lowest. For all methods, there was a positive correlation between the number of resets and the test suite length and between the test case length and the fault detection ratio. Conclusion: The recent methods rely on fewer and longer test cases to reduce the overall test suite length, while the traditional methods produce more and shorter test cases. Longer test cases are correlated to fault detection ratio which favored SPY, though all methods have a ratio of over 92%. (C) 2013 Elsevier B.V. All rights reserved. (AU)

• FAPESP's process: 09/01486-9 - Investigation of finite state machine based testing applied to web services
• Grantee: André Takeshi Endo
• Support Opportunities: Scholarships in Brazil - Doctorate
• FAPESP's process: 12/02232-3 - Input/output transition systems: support to test case generation
• Grantee: Adenilso da Silva Simão
• Support Opportunities: Regular Research Grants