Ambient-awareness in agriculture: 3-D structure and reasoning in the crop field (AACr3)

• Grant number: 17/19282-7
• Support Opportunities: Research Grants - Research Partnership for Technological Innovation - PITE
• Start date: March 01, 2019
• End date: October 31, 2021
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computer Systems
• Agreement: IBM Brasil
• Principal Investigator: Thiago Teixeira Santos
• Grantee: Thiago Teixeira Santos
• Host Institution: Embrapa Informática Agropecuária. Empresa Brasileira de Pesquisa Agropecuária (EMBRAPA). Campinas , SP, Brazil
• Company: IBM Brasil - Indústria, Máquinas e Serviços Ltda
• City: Campinas
• Partner institutions: Empresa Brasileira de Pesquisa Agropecuária (EMBRAPA). Embrapa Instrumentação Agropecuária
• Pesquisadores principais: Eric Rohmer
• Associated researchers: Celina Maki Takemura ; Jayme Garcia Arnal Barbedo ; João Camargo Neto ; Kleber Xavier Sampaio de Souza ; Leonardo Ribeiro Queirós ; Luciano Vieira Koenigkan ; Luís Henrique Bassoi
• Associated scholarship(s): 19/07863-0 - 3-D scene understanding for agriculture using deep neural networks, BP.IC

Abstract

Automation in industrial settings often relies on structured or semi-structured environments. From robotized industrial pipelines to traffic aware car navigation, structure is employed in some way to provide automated services. However, natural and agricultural environments lack structured data, constraining automation in field to limited vehicle auto-guidance. The acquisition of the environment structure data used to be prohibitive considering agricultural crops are relatively low cost products, but recent advances in computer vision and affordable sensors are changing this scenario. This proposal's goal is the automated recovery of the geo-referenced 3-D structure of crop fields and further detection and classification for objects of interest, such as ground, plants, leaves and fruits, by using state-of-the art techniques in machine vision and reasoning. Preliminary results show that 3-D imaging is a viable and versatile way to grab and evaluate plant states and crop fields. In this proposal, proximal remote sensing will be performed by mobile platforms carrying sensors (cameras and LiDAR scanners). Large sets of visual data will be the input for a multiple-view stereo system that will build 3-D models for the crop field, in the form of 3-D points clouds. Computer vision algorithms and machine learning will be employed to detect and classify objects of interest, and information like plant traits, spatial variation in the crop field and other measurements will be properly estimated from the clouds. Field plots of three different crops (including grain crops and fruit crops) will be sensed and structured at different times, capturing multiple developmental stages. This project will integrate cutting edge technologies on imaging, robotics and computer vision in a complete methodology for the acquisition of the 3-D structure of crop fields, approaching issues in automation and high-performance computing. Machine learning-based methods for the extraction of patterns and features from such data will be developed and evaluated against standard methodologies in agricultural research. (AU)