Bilkent University
Department of Computer Engineering

---

M.S. THESIS PRESENTATION

 

Image Inpainting with Diffusion Models and Generative Adversarial Networks

 

Ahmet Burak Yıldırım
Master Student
(Supervisor: Asst. Prof. Ayşegül Dündar)
Computer Engineering Department
Bilkent University

Abstract: We present two novel approaches to image inpainting, a task that involves erasing unwanted pixels from images and filling them in a semantically consistent and realistic way. The first approach uses natural language input to determine which object to remove from an image. We constructed a dataset named GQA-Inpaint for this task and trained a diffusion-based inpainting model on it, which can remove objects from images based on text prompts. The second approach tackles the challenging task of inverting erased images into StyleGAN’s latent space for realistic inpainting and editing. For this task, we propose learning an encoder and a mixing network to combine encoded features of erased images with StyleGAN’s mapped features from random samples. To achieve diverse inpainting results for the same erased image, we combine the encoded features and randomly sampled style vectors via the mixing network. We compare our methods with different evaluation metrics that measure the quality of the models and show significant quantitative and qualitative improvements.

 

DATE: May 20, Monday @ 13:00 Place: EA 409