Deep Learning Techniques Applied to Filter Array Adjustment in Single-Shot Cameras for Multispectral and Hyperspectral Images

• Grant number: 24/08388-2
• Support Opportunities: Scholarships in Brazil - Scientific Initiation
• Start date: August 01, 2024
• End date: September 30, 2025
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computer Systems
• Principal Investigator: Denis Gustavo Fantinato
• Grantee: Tárik Ponte e Sá
• Host Institution: Faculdade de Engenharia Elétrica e de Computação (FEEC). Universidade Estadual de Campinas (UNICAMP). Campinas , SP, Brazil
• Company: Universidade Estadual de Campinas (UNICAMP). Faculdade de Engenharia Elétrica e de Computação (FEEC)
• Associated research grant: 20/09838-0 - BI0S - Brazilian Institute of Data Science, AP.PCPE

Abstract

Multispectral and hyperspectral imaging (MSI/HSI) have advanced consistently in recent decades. As they allow for more robust acquisition of spectral information, their scopes have expanded beyond remote sensing, now finding applications in medicine, food engineering and preservation of cultural heritage.In general, MSI/HSI cameras possess acquisition mechanisms that are prone to being relatively slow - however, there are recent developments in single-shot acquisition modes, which generalize the mechanism used in most digital RGB cameras to MSI/HSI, by utilizing multispectral filter arrays (MSFAs) and demosaicking algorithms.Despite that, there is not yet a definitive solution for obtaining optimal MSFAs and demosaicking algorithms for a given list of bands. However, techniques based in artificial neural networks have shown promising results for the second task. In this analysis, we propose expanding the role of demosaicking algorithms based on deep networks, to also resolve the first task by making the MSFA an adjustable parameter of the network.For training the network, well-known databases shall be used, such as the CAVE dataset from Columbia University, and the HSI/MSI datasets from TokyoTech, already used for the development of single-shot cameras.