Interpretation and explanation of computer vision classification of carambola ( Averrhoa carambola L. ) according to maturity stage

Author(s)

de Moraes, Ingrid Alves ; Barbon Junior, Sylvio ; Barbin, Douglas Fernandes

• Document type: Journal article
• Source: Food Research International; v. 192, p. 10-pg., 2024-07-31.

Abstract

The classification of carambola, also known as starfruit, according to quality parameters is usually conducted by trained human evaluators through visual inspections. This is a costly and subjective method that can generate high variability in results. As an alternative, computer vision systems (CVS) combined with deep learning (DCVS) techniques have been introduced in the industry as a powerful and an innovative tool for the rapid and noninvasive classification of fruits. However, validating the learning capability and trustworthiness of a DL model, aka black box, to obtain insights can be challenging. To reduce this gap, we propose an integrated eXplainable Artificial Intelligence (XAI) method for the classification of carambolas at different maturity stages. We compared two Residual Neural Networks (ResNet) and Visual Transformers (ViT) to identify the image regions that are enhanced by a Random Forest (RF) model, with the aim of providing more detailed information at the feature level for classifying the maturity stage. Changes in fruit colour and physicochemical data throughout the maturity stages were analysed, and the influence of these parameters on the maturity stages was evaluated using the Gradient-weighted Class Activation Mapping (Grad-CAM), the Attention Maps using RF importance. The proposed approach provides a visualization and description of the most important regions that led to the model decision, in wide visualization follows the models an importance features from RF. Our approach has promising potential for standardized and rapid carambolas classification, achieving 91 % accuracy with ResNet and 95 % with ViT, with potential application for other fruits. (AU)

• FAPESP's process: 19/27354-3 - Architecture of colloidal delivery systems: what is the role of structure on the digestibility?
• Grantee: Rosiane Lopes da Cunha
• Support Opportunities: Research Projects - Thematic Grants
• FAPESP's process: 23/07385-7 - Applications of artificial vision for quality monitoring of emerging foods
• Grantee: Douglas Fernandes Barbin
• Support Opportunities: Regular Research Grants
• FAPESP's process: 15/24351-2 - Applications of image analyses and NIR spectroscopy for quality assessment and authentication of food products
• Grantee: Douglas Fernandes Barbin
• Support Opportunities: Research Grants - Young Investigators Grants