I'm trying to build a decoder version of ResNet, i.e. one that goes from the prelogits layer and attempts to recreate the image. I can get it working by using transposed convolutions, but the quality of reconstructed images isn't that great (checkerboard artifacts). I've read that the state of the art is to use UpSampling (i.e. resize) followed by a convolutional layer instead. Whenever I try this my network no longer converges and the images produces are all white/black.
What am I missing?