Improving the pix2pix Architecture for Sketch to Image Translation on a Dataset of Sketches of People to Photos of People

Question

For a university project, I need to create a neural network that translates sketches of people into images. In order to implement such a neural network, I decided to implement a pix2pix GAN architecture. The neural network is trained and evaluated on a modified version of the CUFS dataset provided by my professors.

While the neural network is able to perform image translations, as shown in the image pair below, I wonder if there are ways to improve the quality of the results? In particular, I was wondering if there are ways to make the image look sharper?

I am already augmenting the relatively small training dataset of 70 sketch-photo pairs to 770 ones. Furthermore, I tried to decrease the value of the $\lambda$ parameter. Regarding the epoch, I figured out that the validation loss does not drop significantly after 160 iterations anymore.

I implemented the network in a Jupyter notebook hosted on Google Colab. As you can see in the notebook's last call, the currently achieved L1 loss on the testing dataset is about 0.191 on average. Here is a link to the dataset used for training and testing.

Please feel free to take a look at the code and tell me ways on how to improve the code. In addition, I would be most grateful if you could provide additional ways on how to improve the generated image quality.

Answer 1

I managed to obtain better results with less artifacts by implementing the code of the paper "High-Quality Facial Photo-Sketch Synthesis Using Multi-Adversarial Networks".

I found this code to be quite nice working for my purposes. If you need, I can also provide a Colab notebook with my ported code.