Denoising autoencoders will be your bet in this case. I don't have a book handy for this case. They are good at reconstruction and calculate a good latent representation. Just replace your missing feature with the mean or a fixed value.
CNNs work because of those zeros (the zeros create the boundary on which the change is values is the highest i.e. what networks learn). They are not the problem. Look at your regularizing your network (use dropout, reduce filters if overfitting or increase if underfitting).
The differentiation of shape starts in the initial layers. Colour information is exploited in further layers. Your network has less layers, try increasing your layers (keep filters low in the initial layers).
Also, if you are looking at similarity models, look at contrastive training or ArcFace loss or siamese type models. If you want to stick with ...