3D Sketching using Multi-View Deep Volumetric Prediction



Johanna Delanoy
Mathieu Aubry
Phillip Isola
Alexei A. Efros
Adrien Bousseau

In I3D 2018

[Paper]



Our sketch-based modeling system can process as little as a single perspective drawing (a) to predict a volumetric object (b). Users can refine this prediction and complete it with novel parts by providing additional drawings from other viewpoints (c). This iterative sketching workflow allows quick 3D concept exploration and rapid prototyping (d).

Abstract

Sketch-based modeling strives to bring the ease and immediacy of drawing to the 3D world. However, while drawings are easy for humans to create, they are very challenging for computers to interpret due to their sparsity and ambiguity. We propose a data-driven approach that tackles this challenge by learning to reconstruct 3D shapes from one or more drawings. At the core of our approach is a deep convolutional neural network (CNN) that predicts occupancy of a voxel grid from a line drawing. This CNN provides an initial 3D reconstruction as soon as the user completes a single drawing of the desired shape. We complement this single-view network with an updater CNN that refines an existing prediction given a new drawing of the shape created from a novel viewpoint. A key advantage of our approach is that we can apply the updater iteratively to fuse information from an arbitrary number of viewpoints, without requiring explicit stroke correspondences between the drawings. We train both CNNs by rendering synthetic contour drawings from hand-modeled shape collections as well as from procedurally-generated abstract shapes. Finally, we integrate our CNNs in an interactive modeling system that allows users to seamlessly draw an object, rotate it to see its 3D reconstruction, and refine it by re-drawing from another vantage point using the 3D reconstruction as guidance.

Procedural dataset

Download our procedural dataset here: it contains 20K objects in .off format (in which 60 are separated for testing).

Generate new procedural shapes by using our code for object generation: escad_project.zip

Deep Networks

All our networks are implemented using Caffe framework. To use our models, you will need our fork from the main framework which contains input layers to load and rotate voxel grids: https://github.com/jdelanoy/caffe.

Download our caffe model and trained weights here.