Scene designer: compositional sketch-based image retrieval with contrastive learning and an auxiliary synthesis task

Author(s)

Sampaio Ferraz Ribeiro, Leo ; Bui, Tu ; Collomosse, John ; Ponti, Moacir

• Document type: Journal article
• Source: MULTIMEDIA TOOLS AND APPLICATIONS; v. N/A, p. 23-pg., 2022-12-20.

Abstract

Scene Designer is a novel method for Compositional Sketch-based Image Retrieval (CSBIR) that combines semantic layout synthesis with its main task both to boost performance and enable new creative workflows. While most studies on sketch focus on single-object retrieval, we look to multi-object scenes instead for increased query specificity and flexibility. Our training protocol improves contrastive learning by synthesising harder negative samples and introduces a layout synthesis task that further improves the semantic scene representations. We show that our object-oriented graph neural network (GNN) more than doubles the current SoTA recall@1 on the SketchyCOCO CSBIR benchmark under our novel contrastive learning setting and combined search and synthesis tasks. Furthermore, we introduce the first large-scale sketched scene dataset and benchmark in QuickDrawCOCO. (AU)

• FAPESP's process: 19/02808-1 - Finding correspondences between cross-domain representations using unsupervised feature matching
• Grantee: Leo Sampaio Ferraz Ribeiro
• Support Opportunities: Scholarships abroad - Research Internship - Doctorate (Direct)
• FAPESP's process: 17/22366-8 - Generative networks and feature learning for cross domain visual search
• Grantee: Leo Sampaio Ferraz Ribeiro
• Support Opportunities: Scholarships in Brazil - Doctorate (Direct)
• FAPESP's process: 19/07316-0 - Singularity theory and its applications to differential geometry, differential equations and computer vision
• Grantee: Farid Tari
• Support Opportunities: Research Projects - Thematic Grants