Adaptations to the Heuristic Evaluation (HE) method for novice evaluators

Author(s)

André de Lima Salgado

• Document type: Master's Dissertation
• Press: São Carlos.
• Institution: Universidade de São Paulo (USP). Instituto de Ciências Matemáticas e de Computação (ICMC/SB)
• Defense date: 2017-08-02

Examining board members

Renata Pontin de Mattos Fortes; Lucia Vilela Leite Filgueiras; André Pimenta Freire; Rudinei Goularte

• Advisor: Renata Pontin de Mattos Fortes

Abstract

Heuristic Evaluation (HE) is a popular method of usability inspection. However, its outcomes are dependent on the expertise of evaluators. This study explored and described the difference in quality of outcomes (reports) of a collaborative HE conducted by evaluator groups of distinct composition, regarding different numbers of expert evaluators in each group. Twenty-seven (27) evaluators voluntarily contributed with this study, nine (9) expert and 18 novice evaluators. Thus, I organized seven (7) HE groups according to four (4) different levels of the factor presence of an expert, which ranged from no expert up to three (3) experts in the same group. Each group agreed to provide their reports for this study. Thereafter, I conducted a comparative analysis on the reports based on standard methods of the field and on a cluster analysis of similarities. I described the F-measure for each group report according to a relaxed and a strict criteria. Also, I described the dendrograms formed from the cluster analysis and the respective similarities indicated by each cluster. The results showed that the quality of reports from collaborative HE conducted by experts and novices together can be more similar to the quality of reports from a traditional HE with multiple expert inspectors (Benchmark Group) then to the quality of reports from a collaborative HE conducted by a group composed only by novice evaluators (Baseline Group). Finally, I discuss additional findings and implications for future studies in the field. (AU)

• FAPESP's process: 15/09493-5 - Adaptations for the Heuristic Evaluation (HE) method to novice evaluators
• Grantee: André de Lima Salgado
• Support Opportunities: Scholarships in Brazil - Master