Vinícius Trevisan - MSc in Deep Learning

GANs for Image Generation

A review on generative adversarial networks for image generation. Computers & Graphics. Computers & Graphics 2023.

Category: Data Science
Date: June 2023
Link: Read the paper

As part of my studies for the Masters Degree in Computer Science, my colleagues and I published a review on GANs for image generation. Our publication serves as a comprehensive review, aimed to beginners, and designed to elucidate the fundamental concepts and key techniques utilized by GAN practitioners in this field.

de Souza, V. L. T., Marques, B. A. D., Batagelo, H. C., & Gois, J. P. (2023). A review on generative adversarial networks for image generation. Computers & Graphics.

Abstract

Generative Adversarial Networks (GANs) are a type of deep learning architecture that uses two networks namely a generator and a discriminator that, by competing against each other, pursue to create realistic but previously unseen samples. They have become a popular research topic in recent years, particularly for image processing and synthesis, leading to many advances and applications in various fields. With the profusion of published works and interest from professionals of different areas, surveys on GANs are necessary, mainly for those who aim starting on this topic. In this work, we cover the basics and notable architectures of GANs, focusing on their applications in image generation. We also discuss how the challenges to be addressed in GANs architectures have been faced, such as mode coverage, stability, convergence, and evaluating image quality using metrics.

Graphical Abstract

GANs Tutorial

FUNDAMENTALS AND CHALLENGES OF GENERATIVE ADVERSARIAL NETWORKS FOR IMAGE-BASED APPLICATIONS

Tutorial on GANs presented on the 35th edition of SIBGRAPI, named Fundamentals and Challenges of Generative Adversarial Networks for Image-based Applications.

Category: Data Science
Date: November 2022
Link: Read the paper
Github Repo

On the 35th edition of SIBGRAPI (Conference on Graphics, Patterns and Images), in the city of Natal - Brazil, I presented a tutorial named FUNDAMENTALS AND CHALLENGES OF GENERATIVE ADVERSARIAL NETWORKS FOR IMAGE-BASED APPLICATIONS.

The goal of the 3-hour long tutorial was to introduce researchers on the fundamentals of Generative Adversarial Networks (GANs) and their applications for image processing, computer vision and computer graphics. As part of the presentation, a hands-on section of coding was made available on a Github repository. The tutorial and paper were developed with the help of my avisors, João Paulo Gois and Bruno Dorta.

The associated paper was later published on the congress publication, available on IEEE Xplore.

Abstract

Significant advances in image-based applications have been achieved in recent years, many of which are arguably due to recent developments in Generative Adversarial Networks (GANs). Although the continuous improvement in the architectures of GAN has significantly increased the quality of synthetic images, this is not without challenges such as training stability and convergence issues, to name a few. In this work, we present the fundamentals and notable architectures of GANs, especially for image-based applications. We also discuss relevant issues such as training problems, diversity generation, and quality assessment (metrics).

SIBGRAPI

GANs for sketch-to-image translation and image synthesis

I studied the use of deep learning and generative adversaial networks (GANs) in tasks related to computer vision, image processing, and computer graphics. The topic of my dissertation was "A study about the use of generative adversarial networks for image contour transformation and image synthesis."

Category: Computer Vision
Date: 2020 - 2022
Link: PDF [PT-BR]

Estudo Sobre o Uso de Redes Adversárias Generativas para Transformação de Contornos em Imagens e Síntese de Imagens

A study about the use of generative adversarial networks for image contour transformation and image synthesis

As part of the studies for my master’s degree, I tested many GANs in different applications. I was limited by computational resources, so I could not experiment with state-of-the-art models such as StyleGANs trained from scratch.

On the first part of the work I studied using Pix2Pix and CycleGAN in sketch-to-image aplications. The framework is available on the corresponding Github Repository.

The second part is a proposal of an autoencoder generator to quickly manipulate and edit images. The experiments were made with many combinations of components (generator, discriminator, loss, mapping networks, etc.) and serves as a guide for future projects. The code is also available on Github.

All the studies and results are on my dissertation (in Portuguese).

Abstract

Significant advances in image processing, computer vision, and computer graphics applications were recently achieved, much due to techniques based on Generative Adversarial Networks (GANs). Among those applications we can cite super resolution, image to image translation, style transfer, picture restoration and image synthesis. In this work we study the use of GANs in contour-to-image translation and face image synthesis applications.

The first task involves transforming car sketches into realistic images, and cartoon sketches into a colorized and texturized version. We analyze and discuss the use of both Pix2Pix and CycleGAN architectures in this task, performing experiments with different parameters and configurations to enhance the quality of the synthetic image.

In the synthesis application, inspired by image manipulation works, we explore how to generate synthetic images of human faces using conditional GANs as autoencoders, so that they can generate latent vectors that allow image manipulation directly on the feature latent space. We tested many combinations of different generators, discriminators, loss functions and training techniques to understand which elements contribute to generate better images.

Results

Here are some results presented on the dissertation.

Sketch-to-image

Sketch-to-image results

Autoencoders

Autoencoder results

3D60 - A 3D 360° Camera

As a graduation project for my degree in Control and Automation Engineering I developed with my group a camera capable of filming in 360° and 3D, and even build a prototype with 180° for proving the concept. Many computer vision and image processing techniques were employed into making this project.

Category: Computer vision
Date: December 2017
Link: PDF [PT-BR]

The entertainment industry has been innovating in delivering new ways to consume media.

With the ever growing capacity of the new processors and memories inside embedded systems, it is now possible to create complex types of media, like slow motion videos, virtual reality, 360° videos or 3D videos.

The purpose of this work is to unite both the 3D recording and 360° recording technologies into one single device, intended to generate a new kind of media that is able to cause a sensation of immersion to the viewer.

Thus, the individual that watches a video made with this camera would be able to see the scenario around him in 360 degrees while still maintaining the sensation of depth caused by 3D imaging techniques.

There are, nowadays, virtual reality glasses able to reproduce images and videos with both effects giving the immersion sensation, but generally these images are artificially built with 3D modelling in a computer. A camera that records images in both 360° and 3D allows that real life images, like shows, parties and other events.

About

A little bit about me.

Portfolio

A little bit of what I like to do.

GANs for Image Generation

Data Science Articles

GANs Tutorial

GANs for sketch-to-image translation and image synthesis

3D60 - A 3D 360° Camera

Publications

My latest publications.

A review on Generative Adversarial Networks for image generation

Fundamentals and Challenges of Generative Adversarial Networks for Image-based Applications