Bilkent University
Department of Computer Engineering

---

CS 590/690 SEMINAR

 

Reference-Based 3D-Aware Image Editing with Triplanes

 

Bahri Batuhan Bilecen
Master Student
(Supervisor: Asst. Prof. Ayşegül Dündar)
Computer Engineering Department
Bilkent University

Abstract: Image editing via GANs, especially StyleGAN, is a well-studied problem. Various solutions have been proposed, such as training image encoders to project real-life images onto StyleGAN's latent spaces (W+) and finding suitable editing directions. Another recent development in GANs is the 3D-aware GANs with efficient triplane-based architectures, such as EG3D, where the same W+ methods of StyleGAN are reused to perform edits. However; in this study, we discover that EG3D enables a new editing space, namely triplanes, which achieves reference based edits unattainable via conventional W+ editing methods. Our 3D-aware editing pipeline exploits the differentiability of the volumetric neural renderer, and generates localized high-dimensional region-of-interest masks in triplane space from 2D pixel space via backpropagation. In addition, it includes novel post-processing and fine-tuning stages for the triplane masks and image encoders, respectively, for seamless and realistic image editing. Our method works on cartoon, animal, and real human portraits without additional supervision, shows major improvements over 2D reference-based and W+ editing methods, and achieves state-of-the-art results.

 

DATE: March 04, Monday @ 14:10 Place: EA 502