Static and Dynamic Comparison of Mutation Testing Tools for Python

Author(s)

Guerino, Lucca Renato ; Kuroishi, Pedro Henrique ; Ramada Paiva, Ana Cristina ; Rizzo Vincenzi, Auri Marcelo

• Document type: Journal article
• Source: 23TH BRAZILIAN SYMPOSIUM ON SOFTWARE QUALITY, SBQS 2024; v. N/A, p. 11-pg., 2024-01-01.

Abstract

Context: Mutation testing is a rigorous approach for assessing the quality of test suites by injecting faults (i.e., mutants) into software under test. Tools, such as CosmicRay and Mutpy, are examples of Mutation Testing tools for Python software programs. Problem: With different Python mutation testing tools, comparative analysis is lacking to evaluate their effectiveness in different usage scenarios. Furthermore, the evolution of these tools makes continuous evaluation of their functionalities and characteristics necessary. Method: In this work, we evaluate (statically and dynamically) four Python mutation testing tools, namely CosmicRay, MutPy, MutMut, and Mutatest. In static evaluation, we introduce a comparison framework, adapted from one previously applied to Java tools, and collected information from tool documentation and developer surveys. For dynamic evaluation, we use tests built based on those produced by Pynguin, which are improved through the application of Large Language Models (LLMs) and manual analyses. Then, the adequate test suites were cross-tested among different tools to evaluate their effectiveness in killing mutants each other. Results: Our findings reveal that CosmicRay offers superior functionalities and customization options for mutant generation compared to its counterparts. Although CosmicRay's performance was slightly lower than MutPy in the dynamic tests, its recent updates and active community support highlight its potential for future enhancements. Cross-examination of the test suites further shows that mutation scores varied narrowly among tools, with a slight emphasis on MutPy as the most effective mutant fault model. (AU)

• FAPESP's process: 23/00001-9 - Study and evaluation of mutation test parameterization alternatives aiming at its use in software systems production environments
• Grantee: Auri Marcelo Rizzo Vincenzi
• Support Opportunities: Scholarships abroad - Research
• FAPESP's process: 19/23160-0 - Mutation-based software testing with high efficiency and low technical debt: automated process and free support environment prototype
• Grantee: Auri Marcelo Rizzo Vincenzi
• Support Opportunities: Regular Research Grants