Multi-modal Real-time Weed Classification and Localization in Deep Neural Network Frameworks Validated on Rasberry-Pi

• Grant number: 25/24078-6
• Support Opportunities: Research Grants - Visiting Researcher Grant - International
• Start date: June 01, 2026
• End date: July 26, 2026
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
• Principal Investigator: Zhao Liang
• Grantee: Zhao Liang
• Visiting researcher: Karan Nathwani
• Visiting researcher institution: Indian Institute Of Technology, Jammu/Iit , India
• Host Institution: Faculdade de Filosofia, Ciências e Letras de Ribeirão Preto (FFCLRP). Universidade de São Paulo (USP). Ribeirão Preto , SP, Brazil
• City of the host institution: Ribeirão Preto
• Associated research grant: 24/15430-5 - Machine learning-based multi-modal data fusion and growth modeling for soybean production improvement, AP.PFPMCG.TEM

Abstract

Our project proposes setting up an innovative IoT-based system in real-time for weed detection, targeted mainly for the precision in soybean culture. The system integrates UAV-based RGB cameras and multi-modal data processing considering also multi and hyper-spectral images with different deep learning models for automatic detection, classification, and localization of weed species, mainly herbicide-resistant weeds such as hairy fleabane (Conyza spp.) and sourgrass (Digitaria insularis), which are globally abundant in soybean culture.The intent of the project is to develop a comprehensive solution that might take aerial images at high resolution and at the standing altitude, handle the processing side of these data in object detection algorithms and generate herbicide application maps that can be applied to site-specific weed control. It is envisioned that by minimizing herbicide usage through targeted application, the system would be a minus production cost, thus encouraging environment preservation and sustainable agriculture.Key deliverables include a novel annotated dataset (which will be collected by UAVs having RGB, multi and hyper-spectral images which will be manually annotated by professionals), optimized deep learning models achieving precise, real-time processing capability, and a complete IoT framework for automated herbicide mapping, thus rendering a ready-to-deploy solution for application in large-scale agricultural operations worldwide, with the potential of economic and environmental advantage in precision agriculture. (AU)