On Applying Metamorphic Testing: An Empirical Study On Academic Search Engines

Author(s)

Andrade, Stevao ; Santos, Italo ; Brito Junior, Claudinei ; Junior, Misael ; de Souza, Simone R. S. ; Delamaro, Marcio E. ; IEEE

• Document type: Journal article
• Source: 2019 IEEE/ACM 4TH INTERNATIONAL WORKSHOP ON METAMORPHIC TESTING (MET 2019); v. N/A, p. 8-pg., 2019-01-01.

Abstract

Software testing can be a helpful practice to certify the quality of a product. However, there are programs which are hard, if not impossible, to determine the expected outputs. This problem is called the "oracle problem". Metamorphic testing (MT) is an approach that aims to alleviate this problem by setting a series of relations, called metamorphic relations (MRs). This paper applies MT through a user-oriented approach and the following MRs: MPublished, MPTitle, MPShuffleJD and Top1Absent as a strategy for evaluating, verifying, and validating four Academic Search Engines (ASEs): ACM, IEEE, ScienceDirect, and Springer. Therefore, we conducted an experimental study to analyze how MRs can contribute to verifying the correctness of the properties of ASEs. Results indicate that the ASEs have performed differently in their execution for each MR analyzed. This shows that the algorithms of the scientific search engines have different behaviors among the characteristics analyzed. These behaviors may not be properly characterized by a possible failure, but by a simple design choice. On the other hand, the approach is useful for search engine users that need to be sure that the ASEs behave as expected during a search, as the results of secondary studies strongly depend on the correct behavior of the scientific search engine used. (AU)

• FAPESP's process: 17/19492-1 - Mutation testing applied to virtual reality environments modeled with scene graph
• Grantee: Stevão Alves de Andrade
• Support Opportunities: Scholarships in Brazil - Doctorate
• FAPESP's process: 18/10183-9 - Study and definition of project attributes for selection of testing techniques for concurrent software
• Grantee: Italo de Oliveira Santos
• Support Opportunities: Scholarships in Brazil - Master