Semantic localization and mapping by proximal multimodal sensing for agricultural automation

• Grant number: 24/10267-9
• Support Opportunities: Scholarships in Brazil - Post-Doctoral
• Start date: May 01, 2025
• End date: April 30, 2027
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computing Methodologies and Techniques
• Principal Investigator: Thiago Teixeira Santos
• Grantee: Dheeraj Bharti
• Host Institution: Embrapa Agricultura Digital. Empresa Brasileira de Pesquisa Agropecuária (EMBRAPA). Campinas , SP, Brazil
• Associated research grant: 22/09319-9 - Center of Science for Development in Digital Agriculture - CCD-AD/SemeAr, AP.CCD

Abstract

The lack of automation in fruit growing and perennial crops is a significant issue in agriculture, especially considering the diminishing availability of labor. Fruit farming is a labor-intensive activity, particularly in tasks such as pruning, tying, and harvesting. Farmers often organize collective efforts to meet the high demand for labor during peak seasons. This is especially true for family farmers, whose operations rely heavily on family labor and cooperation with neighbors. The scarcity of labor poses a significant challenge for these farmers and the fruit supply chain, highlighting the need for automation to address labor shortages and alleviate the harsh working conditions in the field. This issue has been identified in fruit-growing districts (DATs) by the Instituto de Economia Agrícola through the Semear Project, particularly in the Vacaria (RS) and São Miguel Arcanjo (SP) districts.Unlike fruit production for industrial purposes, such as juice production, fruit growing for the fresh food consumer market requires careful handling of fruits, as even small mechanical damage can significantly reduce selling prices for farmers. The combination of automation in orchards, which are outdoor and semi-structured environments, with the need for precise manipulation of crops, points to the need for machines capable of performing semantic localization and mapping of orchards. Simultaneous localization and mapping, known in the literature on robotics and computer vision as SLAM, is the ability of an autonomous agent to map an unknown environment while simultaneously localizing itself within that map. The semantic component involves dividing the map into meaningful elements, such as trees, fruits, poles, grass, dirt, etc., which is essential for tasks like vision-based yield mapping and automated harvesting.Despite growing interest in agricultural SLAM, there is a lack of public datasets and benchmarks for agricultural settings, unlike the urban scenarios driven by autonomous vehicle research. The present research plan aims to narrow this gap by (i) developing a multimodal sensing module for large-scale data collection in orchards, (ii) creating a public dataset for semantic SLAM research in orchard settings, and (iii) establishing a benchmark for the evaluation and comparison of SLAM systems in such environments.