Modernisme meets StyleGAN

I've spent some time training a StyleGAN2 model on architectural elements, specifically, I took into account only photos of doors mostly, fences, and windows from Barcelona, which represent the Catalan modernism movement. Here are some results from the generative adversarial network and some experimentation with model interpolation.

Why Modernisme?

Movement of architecture at the juncture of 19th and 20th centuries, centred in the city of Barcelona. Gaudí first comes to mind. Loosely speaking, Catalan modernism, or Modernisme for short, is an equivalent to Art Nouveau in France – the natural world was a central inspiration in this movement: sinuous, organic lines, use of nature.

Therefore, creating a Generative adversarial network that will be able to detect typical elements and main features of the Modernism movement is a great goal.

If the photo of the Paris Métro entrance rings a bell, you're all set.

First Things First – Dataset

Finding and preparing the dataset is the most challenging part of experiments with neural networks. For my project, I parsed roughly 5k photos using Flickr API, which is a relatively small dataset, given that pre-trained StyleGAN model used about 70k.

An example of images that I used for training.

Preprocessing

It is rumored that Data Scientists spend most of their time cleaning and preparing data rather than building models – very true.

My preprocessing pipeline for this dataset consists of the following steps:

1. Object detection with OpenCV (doors or quadrangles in our case).
2. Image Alignment using Homography.
3. Cropping.
4. Splitting items with 1:3 aspect ratio (like the narrow doors above) into three squared photos.
5. Image Augmentation, including shifts, shear, and flips.

These simple steps help me to increase the dataset to the 70k benchmark mentioned above.

Due to the limitation of my cloud resources – a single GPU (Tesla V100 NVIDIA), I resized my dataset to 256*256.

Basic Door Anatomy

This section is more for non-native English speakers like me to be more familiar with the terminology. The body of any door is essentially made of three parts: stiles, rails, and panels.

Training networks

I used the official TensorFlow Implementation of StyleGAN2.

To reduce memory consumption, I decreased the number of channels in the generator and discriminator, latent size, and batch size.

The training process took about 40 hours.

Results

The monochrome cover photo of this post shows the results of GAN model; here is the set of artificial doors in color.

As you can see, the model perfectly catches the proportions, ratios between rails and panels, key elements of doors. There are also organic lines in patterns, so the model represents the creativity of Art Nouveau architecture. AI Modernism, isn't it?

I did interpolation in StyleGAN's space with latent walk approach to get this GIF.

This is it! Let's call it DoorsStyleGAN. You can download my pre-trained model (.pkl file) from this link. Use it for Transfer Learning, Style mixing, CycleGAN, or for any other purposes.

Shoot a comment below if you're in need of Google Colab notebook implementation. UPDATE: Here it is.