Vision for the blind: translating 3D visual concepts into 3D auditory clues

• Grant number: 13/21349-1
• Support Opportunities: Scholarships in Brazil - Master
• Start date: November 01, 2013
• End date: July 31, 2014
• Field of knowledge: Physical Sciences and Mathematics - Computer Science - Computer Systems
• Agreement: Microsoft Research
• Principal Investigator: Luiz César Martini
• Grantee: Felipe Leonel Grijalva Arévalo
• Host Institution: Instituto de Computação (IC). Universidade Estadual de Campinas (UNICAMP). Campinas , SP, Brazil
• Company: Universidade Estadual de Campinas (UNICAMP). Instituto de Computação (IC)
• Associated research grant: 12/50468-6 - Vision for the blind: translating 3D visual concepts into 3D auditory clues, AP.PITE

Abstract

The goal of this project is to construct and validate a complete proof-of-concept assistive device for the blind and low-vision. The device is based on translating visual information into auditory information. The key problem in translating visual information into auditory is one of bandwidth, which is order of magnitudes higher in the visual system when compared to the auditory system. We believe this is, in essence, what has made most of the previous sensory substitution proposals fail. In this project, we propose to circumvent that by using two key concepts: 1) using computer vision to simplify the visual scene, and 2) using 3D audio to exploit the inherent special sense of the auditory system. This system will use computer vision algorithms to extract high-level information and will communicate this information using different codification approaches, but exploring 3D audio capabilities to provide spatial localization. The hardware component of this system will combine an off-the-shelf image + depth camera (Microsoft Kinect), and accelerometer/gyroscope, a headphone, and a notebook. The software component will be modular and extensible. The system will have distinct modes of operations, to provide specialized functionalities such as navigation, people localization, and textual information translation, such as signs and currency identification. Each of these modes has different requirements - in the computer vision side to extract the desired high-level information of the environment, and in the 3D audio to best communicate the desired information. A fully operational system presents significant scientific and technological challenges, from the development and proper integration of computer vision algorithms to the best design and user - validation of the audio interfaces. (AU)

Scientific publications

(References retrieved automatically from Web of Science and SciELO through information on FAPESP grants and their corresponding numbers as mentioned in the publications by the authors)

GRIJALVA, FELIPE; MARTINI, LUIZ CESAR; MASIERO, BRUNO; GOLDENSTEIN, SIOME. A Recommender System for Improving Median Plane Sound Localization Performance Based on a Nonlinear Representation of HRTFs. IEEE ACCESS, v. 6, p. 24829-24836, 2018. (12/50468-6, 14/14630-9, 13/21349-1)

GRIJALVA, FELIPE; MARTINI, LUIZ CESAR; FLORENCIO, DINEI; GOLDENSTEIN, SIOME. Interpolation of Head-Related Transfer Functions Using Manifold Learning. IEEE SIGNAL PROCESSING LETTERS, v. 24, n. 2, p. 221-225, FEB 2017. (12/50468-6, 13/21349-1, 14/14630-9)

NETO, LAURINDO BRITTO; GRIJALVA, FELIPE; MARGARETH LIMA MAIKE, VANESSA REGINA; MARTINI, LUIZ CESAR; FLORENCIO, DINEI; CALANI BARANAUSKAS, MARIA CECILIA; ROCHA, ANDERSON; GOLDENSTEIN, SIOME. A Kinect-Based Wearable Face Recognition System to Aid Visually Impaired Users. IEEE TRANSACTIONS ON HUMAN-MACHINE SYSTEMS, v. 47, n. 1, p. 52-64, FEB 2017. (12/50468-6, 15/19222-9, 13/21349-1, 14/14630-9)

Academic Publications

(References retrieved automatically from State of São Paulo Research Institutions)

ARÉVALO, Felipe Leonel Grijalva. Dimensionality reduction using Isomap applied to spatial audio. 2014. Master's Dissertation - Universidade Estadual de Campinas (UNICAMP). Faculdade de Engenharia Elétrica e de Computação Campinas, SP.