Graph Based Language Model for Link Prediction

• Grant number: 24/10001-9
• Support Opportunities: Scholarships in Brazil - Doctorate
• Start date: March 01, 2025
• End date: June 30, 2028
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
• Principal Investigator: Ricardo Marcondes Marcacini
• Grantee: Kenzo Miranda Sakiyama
• Host Institution: Instituto de Ciências Matemáticas e de Computação (ICMC). Universidade de São Paulo (USP). São Carlos , SP, Brazil

Abstract

Events are characterized by descriptions containing places, people involved, times, etc., and their monitoring is of great interest for understanding real-world events. To mine information from events, using machine learning and deep learning techniques, they are often represented through graphs in which their vertices and edges correspond to the components and relationships of the events, respectively. Within this context, the use of random walks for mining information in graphs is not new in the literature and has already proven to be quite effective. Particularly, the combination of walks with language models, such as word2vec, has established itself as a popular approach for solving graph tasks, following the example of deepwalk and node2vec. Considering this use of random walks, there are uninvestigated gaps in the literature. First, word2vec models operate over small contexts and in a less comprehensive way than recent language models. Secondly, new language models emerged after word2vec, such as models based on BERT and GPT, and, to date, there are few works in the literature that explore them with random walks on graphs. Therefore, the objective of this proposal is to investigate the use of language models based on Transformers, in a set of random walks, aiming at graph modeling and model evaluation in the link prediction task, which allow analyzing the relationship between events and its components of places, people and organizations. Different variations of the architecture will be investigated, as well as graph walking techniques, and they will be compared using the aforementioned task. Furthermore, the proposal will be compared with the current state of the art in the task in order to empirically prove its effectiveness. In order to evaluate the methodology in a real problem, it will also be evaluated in predicting relationships in different event bases.