Hyperparameter Tuning in Artificial Neural Networks for Mitigating Nonlinearities in Coherent Optical Communication Systems

• Grant number: 24/17085-3
• Support Opportunities: Scholarships in Brazil - Scientific Initiation
• Start date: February 01, 2025
• End date: January 31, 2026
• Field of knowledge: Engineering - Electrical Engineering - Telecommunications
• Principal Investigator: Ivan Aritz Aldaya Garde
• Grantee: Gabriel Bozelli Dias
• Host Institution: Faculdade de Engenharia. Universidade Estadual Paulista (UNESP). Campus São João da Boa Vista. São João da Boa Vista , SP, Brazil
• Associated research grant: 24/01712-9 - Compensation for transmission impairments in advanced optical communication systems employing artificial intelligence, AP.R

Abstract

With the increase in internet data transmission rates, coherent digital optical systems have needed optimization to achieve better performance. Nonlinear effects, such as the Kerr effect in fiber, are the main limiting factors in these systems, and over the years, various tools have been developed to mitigate these effects. Artificial neural networks (ANNs) have emerged as an attractive model for this task due to their low computational cost. However, a major challenge in their implementation is finding the optimal configuration of their hyperparameters. This work proposes the use of a genetic algorithm to find the best configuration of a neural network to mitigate the nonlinear effects in an optical system and achieve a lower bit error rate.