Photorealistic rendering of painting + Venice Underwater

Abstract

The main goal of our project is to transform old paintings and drawings of Venice into a photorealistic representation of the past, thus create photos never taken. Therefore we will test different kinds of GANs (Generative Adversarial Networks, e.g. CycleGAN, BigGAN, PGGAN, Contrastive-unpaired-translation (CUT)) and different kinds of loss functions (e.g. EdgeLoss). They will be trained on a dataset of real photos of Venice together with a dataset of either b/w drawings, or b/w paintings, or coloured paintings. Finally, the trained models should take a drawing or painting as input and deliver its photorealistic representation as output. As a subgoal of the project, another GAN will be trained to create a visual representation of Venice as an underwater city. This representation takes the rising sea level into account and allows to scale time into the future by visualizing the drowning (/ declining?) of the world heritage.

Planning

Week	Tasks
9.11.2020 - 15.11.2020 (week 9)	Midterm presentation ✔️ Test edge loss ✔️ Prepare sweep visualization ✔️
16.11.2020 - 22.11.2020 (week 10)	Test edge loss (cont) ✔️ Prepare project Wiki page 💬 Perform sweep (1) (params: lambda_GAN = [0.8, 1.0, 1.2], lambda_NCE = [0.8, 1.0, 1.2]) ✔️ Perform sweep (2) (params: edgeLoss = [0.8, 1.6], canny1 = [150, 250], canny2 = [250, 400]) ✔️ Implement spectral loss 💬 Clean colour photo data ✔️
23.11.2020 - 29.11.2020 (week 11)	Perform sweep (3) (params: n_epochs = [100, 250], n_epochs_decay = [100, 250]) Test best performing models from sweeps (1), (2), (3) Test self-attention model Clean underwater picture data Test underwater model
30.11.2020 - 6.12.2020 (week 12)	Finish tests, select best hyperparameters/additions for each model Monochrome to photo Colour to photo Photo to underwater (hopefully)) Clean up code Implement front end Write report
7.12.2020 - 13.12.2020 (week 13)	Finish writing report Prepare presentation
14.12.2020 - 16.12.2020 (week 14)	Final project presentation

Challenges

Data collection

• Need more data than expected (black and white drawings, black and white paintings, coloured paintings, underwater data, Venice photos, Venice landmark photos)
• Need to clean more data than expected (removing watermarks, borders, removing unusual images)
• Content in input and output images does not always match. For example:
  • Large dense crowds in paintings that do not appear in modern day photos
  • Old boat structures vs new boat structures
• Collecting good quality underwater data that contains details of underwater structures, rather than just images of algae and pond scum 🤿

Model challenges

• Long training time (~20 hours for 400 epochs on one GPU)
• Many hyperparameters to tune: we must carefully select dependent hyperparameters to test model performance on each sweep
• The CUT model requires more images than described in the paper