Semantically-Aware Contrastive Learning for multispectral remote sensing images

Author(s)

Stival, Leandro ; Torres, Ricardo da Silva ; Pedrini, Helio

• Document type: Journal article
• Source: ISPRS JOURNAL OF PHOTOGRAMMETRY AND REMOTE SENSING; v. 223, p. 15-pg., 2025-03-18.

Abstract

Satellites continuously capture vast amounts of data daily, including multispectral remote sensing images (MSRSI), which facilitate the analysis of planetary processes and changes. New machine-learning techniques are employed to develop models to identify regions with significant changes, predict land-use conditions, and segment areas of interest. However, these methods often require large volumes of labeled data for effective training, limiting the utilization of captured data in practice. According to current literature, self-supervised learning (SSL) can be effectively applied to learn how to represent MSRSI. This work introduces Semantically Aware Contrastive Learning (SACo+), a novel method for training a model using SSL for MSRSI. Relevant known band combinations are utilized to extract semantic information from the MSRSI and texture-based representations, serving as anchors for constructing a feature space. This approach is resilient against changes and yields semantically informative results using contrastive techniques based on sample visual properties, their categories, and their changes over time. This enables training the model using classic SSL contrastive frameworks, such as MoCo and its remote sensing version, SeCo, while also leveraging intrinsic semantic information. SACo+ generates features for each semantic group (band combination), highlighting regions the images (such as vegetation, urban areas, and water bodies), and explores texture properties encoded based on Local Binary Pattern (LBP). To demonstrate the efficacy of our approach, we trained ResNet models with MSRSI using the semantic band combinations in SSL frameworks. Subsequently, we compared these models three distinct tasks: land cover classification task using the EuroSAT dataset, change detection using the OSCD dataset, and semantic segmentation using the PASTIS and GID datasets. Our results demonstrate that leveraging semantic and texture features enhances the quality of the feature space, leading to improved performance all benchmark tasks. The model implementation and weights are available at https://github.com/lstival/SACo - As of Jan. 2025. (AU)

• FAPESP's process: 23/11556-1 - Novel deep learning methods for remote sensing imagery
• Grantee: Leandro Stival
• Support Opportunities: Scholarships abroad - Research Internship - Doctorate
• FAPESP's process: 22/12294-8 - Convolutional Networks with Attention for Video Color Propagation
• Grantee: Leandro Stival
• Support Opportunities: Scholarships in Brazil - Doctorate