SILA: a system for scientific image analysis

Author(s):Show less -

Moreira, Daniel ; Cardenuto, Joao Phillipe ; Shao, Ruiting ; Baireddy, Sriram ; Cozzolino, Davide ; Gragnaniello, Diego ; Abd-Almageed, Wael ; Bestagini, Paolo ; Tubaro, Stefano ; Rocha, Anderson ; Scheirer, Walter ; Verdoliva, Luisa ; Delp, Edward

• Document type: Journal article
• Source: SCIENTIFIC REPORTS; v. 12, n. 1, p. 15-pg., 2022-10-31.

Abstract

A great deal of the images found in scientific publications are retouched, reused, or composed to enhance the quality of the presentation. In most instances, these edits are benign and help the reader better understand the material in a paper. However, some edits are instances of scientific misconduct and undermine the integrity of the presented research. Determining the legitimacy of edits made to scientific images is an open problem that no current technology can perform satisfactorily in a fully automated fashion. It thus remains up to human experts to inspect images as part of the peer-review process. Nonetheless, image analysis technologies promise to become helpful to experts to perform such an essential yet arduous task. Therefore, we introduce SILA, a system that makes image analysis tools available to reviewers and editors in a principled way. Further, SILA is the first human-in-the-loop end-to-end system that starts by processing article PDF files, performs image manipulation detection on the automatically extracted figures, and ends with image provenance graphs expressing the relationships between the images in question, to explain potential problems. To assess its efficacy, we introduce a dataset of scientific papers from around the globe containing annotated image manipulations and inadvertent reuse, which can serve as a benchmark for the problem at hand. Qualitative and quantitative results of the system are described using this dataset. (AU)

• FAPESP's process: 17/12646-3 - Déjà vu: feature-space-time coherence from heterogeneous data for media integrity analytics and interpretation of events
• Grantee: Anderson de Rezende Rocha
• Support Opportunities: Research Projects - Thematic Grants