Static and Dynamic Comparison of Mutation Testing Tools for Python

Autor(es)

Guerino, Lucca Renato ; Kuroishi, Pedro Henrique ; Ramada Paiva, Ana Cristina ; Rizzo Vincenzi, Auri Marcelo

• Tipo de documento: Artigo Científico
• Fonte: 23TH BRAZILIAN SYMPOSIUM ON SOFTWARE QUALITY, SBQS 2024; v. N/A, p. 11-pg., 2024-01-01.

Resumo

Context: Mutation testing is a rigorous approach for assessing the quality of test suites by injecting faults (i.e., mutants) into software under test. Tools, such as CosmicRay and Mutpy, are examples of Mutation Testing tools for Python software programs. Problem: With different Python mutation testing tools, comparative analysis is lacking to evaluate their effectiveness in different usage scenarios. Furthermore, the evolution of these tools makes continuous evaluation of their functionalities and characteristics necessary. Method: In this work, we evaluate (statically and dynamically) four Python mutation testing tools, namely CosmicRay, MutPy, MutMut, and Mutatest. In static evaluation, we introduce a comparison framework, adapted from one previously applied to Java tools, and collected information from tool documentation and developer surveys. For dynamic evaluation, we use tests built based on those produced by Pynguin, which are improved through the application of Large Language Models (LLMs) and manual analyses. Then, the adequate test suites were cross-tested among different tools to evaluate their effectiveness in killing mutants each other. Results: Our findings reveal that CosmicRay offers superior functionalities and customization options for mutant generation compared to its counterparts. Although CosmicRay's performance was slightly lower than MutPy in the dynamic tests, its recent updates and active community support highlight its potential for future enhancements. Cross-examination of the test suites further shows that mutation scores varied narrowly among tools, with a slight emphasis on MutPy as the most effective mutant fault model. (AU)

• Processo FAPESP: 23/00001-9 - Estudo e avaliação de alternativas de parametrização do teste de mutação visando o seu uso em ambientes de produção de sistemas de software
• Beneficiário: Auri Marcelo Rizzo Vincenzi
• Modalidade de apoio: Bolsas no Exterior - Pesquisa
• Processo FAPESP: 19/23160-0 - Teste de software baseado em mutação com alta eficácia e baixa dívida técnica: processo automatizado e protótipo de ambiente de apoio livres
• Beneficiário: Auri Marcelo Rizzo Vincenzi
• Modalidade de apoio: Auxílio à Pesquisa - Regular