Development of a system based on multimodal transformers for information retrieval in offshore oil and gas extraction platforms.

• Grant number: 25/16800-3
• Support Opportunities: Scholarships in Brazil - Scientific Initiation
• Start date: October 01, 2025
• End date: September 30, 2026
• Field of knowledge: Engineering - Electrical Engineering
• Principal Investigator: Thiago de Castro Martins
• Grantee: Júlia Yui Aoki
• Host Institution: Escola Politécnica (EP). Universidade de São Paulo (USP). São Paulo , SP, Brazil
• Company: Universidade de São Paulo (USP). Escola Politécnica (EP)
• Associated research grant: 22/03698-8 - OTIC Offshore Technology Innovation Centre, AP.PCPE

Abstract

Multimodal transformer technology has stood out in recent years due to its ability to simultaneously integrate and process data from multiple modalities, such as text, images, audio, and time series. Thanks to technological advances, increasing volumes of multimodal data are being transmitted, and a growing number of multimodal application scenarios are emerging.In this context, multimodal transformers have proven to be promising for complex interpretation and inference tasks involving heterogeneous data. One example of their applicability is presented in a recent study by China Agricultural University, which demonstrates the use of this technology in plant disease detection by simultaneously processing images, textual descriptions, and sensor data-achieving an accuracy of 94%.In offshore platforms, embedded sensors generate data in various formats-graphs, images, and numerical values-as well as textual reports written by the responsible operators. It is essential for the proper operation of these platforms that professionals are attentive to such data and able to interpret it in a timely manner. Late failure detection can lead to significant human and financial losses.However, due to the complexity of modern platforms, the number of required sensors has been increasing, which consequently leads to a greater volume of data to be monitored. Thus, the use of artificial intelligence models based on multimodal transformer architectures becomes a promising approach to assist in the diagnosis of such data.Therefore, the present project aims to support the development of a system based on multimodal transformers for diagnosing data collected from offshore platform sensors. This system can be used by professionals as a tool to retrieve relevant information about the platform. (AU)