I'm trying to build a decoder version of ResNet, i.e. one that goes from the prelogits layer and attempts to recreate the image. I can get it working by using transposed convolutions, but the quality of reconstructed images isn't that great (checkerboard artifacts). I've read that the state of the art is to use UpSampling (i.e. resize) followed by a convolutional layer instead. Whenever I try this my network no longer converges and the images produces are all white/black.
What am I missing?
Presumably, you have the seen the Distil article discussing checkerboard artifacts due to transposed convolutional layers, where they recommend upsampling (resizing) and then doing regular convolution. This is known to be a reasonable approach, so it is likely something specific to your case. Without architectural details, I can only guess that it might be (a) your hyper-parameters have been tuned to work with transposed conv layers, so it breaks with this different approach, or (b) you have introduced some kind of vanishing gradients or convergence issue (e.g., not using residual connections, not using batch norm, too large of a learning rate, etc...). The loss (e.g., $L_2$ reconstruction) and it's gradients may be of a different magnitude than what the other (hyper-)parameters were expecting, for example. Feel free to post back with more info.
