Evaluating test suite characteristics, cost, and effectiveness of FSM-based testing methods

Autor(es)

Endo, Andre Takeshi ^[1] ; Simao, Adenilso ^[1]

Afiliação do(s) autor(es)

^[1] Univ Sao Paulo, Inst Ciencias Matemat & Comp, BR-13560970 Sao Carlos, SP - Brazil

• Tipo de documento: Artigo Científico
• Fonte: INFORMATION AND SOFTWARE TECHNOLOGY; v. 55, n. 6, p. 1045-1062, JUN 2013.
• Citações Web of Science: 12

Resumo

Context: Testing from finite state machines has been investigated due to its well-founded and sound theory as well as its practical application. There has been a recurrent interest in developing methods capable of generating test suites that detect all faults in a given fault domain. However, the proposal of new methods motivates the comparison with traditional methods. Objective: We compare the methods that generate complete test suites from finite states machines. The test suites produced by the W, HSI, H, SPY, and P methods are analyzed in different configurations. Method: Complete and partial machines were randomly generated varying numbers of states, inputs, outputs, and transitions. These different configurations were used to compare test suite characteristics (number of resets, test case length) and the test suite length (i.e., the sum of the length of its test cases). The fault detection ratio was evaluated using mutation testing to produce faulty implementations with an extra state. Results: On average, the recent methods (H, SPY, and P) produced longer test cases but smaller test suites than the traditional methods (W, HSI). The recent methods generated test suites of similar length, though P produced slightly smaller test suites. The SPY and P methods had the highest fault detection ratios and HSI had the lowest. For all methods, there was a positive correlation between the number of resets and the test suite length and between the test case length and the fault detection ratio. Conclusion: The recent methods rely on fewer and longer test cases to reduce the overall test suite length, while the traditional methods produce more and shorter test cases. Longer test cases are correlated to fault detection ratio which favored SPY, though all methods have a ratio of over 92%. (C) 2013 Elsevier B.V. All rights reserved. (AU)

• Processo FAPESP: 09/01486-9 - Investigação de teste baseado em máquinas de estados finitos aplicado a serviços web
• Beneficiário: André Takeshi Endo
• Modalidade de apoio: Bolsas no Brasil - Doutorado
• Processo FAPESP: 12/02232-3 - Sistemas de transição de entrada/saída: apoio para a geração de casos de teste
• Beneficiário: Adenilso da Silva Simão
• Modalidade de apoio: Auxílio à Pesquisa - Regular