Compact descriptors for sketch-based image retrieval using a triplet loss convolutional neural network

Author(s)

Bui, T. ; Ribeiro, L. ; Ponti, M. ; Collomosse, J.

• Document type: Journal article
• Source: COMPUTER VISION AND IMAGE UNDERSTANDING; v. 164, p. 11-pg., 2017-11-01.

Abstract

\We present an efficient representation for sketch based image retrieval (SBIR) derived from a triplet loss convolutional neural network (CNN). We treat SBIR as a cross-domain modelling problem, in which a depiction invariant embedding of sketch and photo data is learned by regression over a siamese CNN architecture with half-shared weights and modified triplet loss function. Uniquely, we demonstrate the ability of our learned image descriptor to generalise beyond the categories of object present in our training data, forming a basis for general cross-category SBIR. We explore appropriate strategies for training, and for deriving a compact image descriptor from the learned representation suitable for indexing data on resource constrained e.g. mobile devices. We show the learned descriptors to outperform state of the art SBIR on the defacto standard Flickrl5k dataset using a significantly more compact (56 bits per image, i. e. approximate to 105KB total) search index than previous methods. Datasets and models are available from the CVSSP datasets server at www.cvssp.org. (C) 2017 Elsevier Inc. All rights reserved. (AU)

• FAPESP's process: 15/26050-0 - Multiple-instance image ranking for sketch-based image retrieval
• Grantee: Leo Sampaio Ferraz Ribeiro
• Support Opportunities: Scholarships abroad - Research Internship - Scientific Initiation
• FAPESP's process: 16/16111-4 - Feature learning applied to sketch-based image retrieval and low-altitude remote sensing
• Grantee: Moacir Antonelli Ponti
• Support Opportunities: Regular Research Grants