Photorealistic rendering of painting + Venice Underwater

Abstract

The main goal of our project is to transform old paintings and drawings of Venice into a photorealistic representation of the past, thus create photos never taken. Therefore we will test different kinds of GANs (Generative Adversarial Networks, e.g. CycleGAN, BigGAN, PGGAN) and different kinds of loss functions (e.g. EdgeLoss). They will be trained on a dataset of real photos of Venice together with a dataset of either b/w drawings, or b/w paintings, or coloured paintings. Finally, the trained models should take a drawing or painting as input and deliver a photorealistic representation as output. As a subgoal of the project, another GAN will be trained to create a visual representation of Venice as an underwater city. This representation takes the rising sea level into account and allows to scale time into the future and to visualize future Venice as a drowning / declining world heritage.

Planning

Week	Tasks	Status
9.11.2020 - 15.11.2020 (week 9)	Midterm presentation Test edge loss Prepare sweep visualization	Done
16.11.2020 - 22.11.2020 (week 10)	Test edge loss (cont) Prepare project Wiki page Perform sweep (1) (params: lambda_GAN = [0.8, 1.0, 1.2], lambda_NCE = [0.8, 1.0, 1.2]) Perform sweep (2) (params: edgeLoss = [0.8, 1.6], canny1 = [150, 250], canny2 = [250, 400]) Implement spectral loss Clean colour photo data
23.11.2020 - 29.11.2020 (week 11)	Perform sweep (3) (params: n_epochs = [100, 250], n_epochs_decay = [100, 250]) Test best performing models from sweeps (1), (2), (3) Test self-attention model Clean underwater picture data Test underwater model
30.11.2020 - 6.12.2020 (week 12)	Finish tests, select best hyperparameters/additions for each model Monochrome to photo Colour to photo Photo to underwater (hopefully)) Clean up code Implement front end Write report
7.12.2020 - 13.12.2020 (week 13)	Finish writing report Prepare presentation
14.12.2020 - 16.12.2020 (week 14)	Final project presentation

Challenges

Data collection

• Need more data than expected (black and white drawings, black and white paintings, coloured paintings, underwater data, Venice photos, Venice landmark photos)
• Need to clean more data than expected (removing watermarks, borders, removing unusual images)
• Content in input and output images does not always match. For example:
  • Large dense crowds in paintings that do not appear in modern day photos
  • Old boat structures vs new boat structures
• Collecting good quality underwater data that contains details of underwater structures, rather than just images of algae and pond scum 🤿

Model challenges

• Long training time (~20 hours for 400 epochs on one GPU)
• Many hyperparameters to tune: we must carefully select dependent hyperparameters to test model performance on each sweep
• The CUT model requires more images than described in the paper