Mean, zero, duplication (i.e., constant extrapolation), reflection, and symmetric padding are all valid and widely used methods of padding for conv layers. To my knowledge, there is no systemic study that says one is better than the other in all cases. In other words, one is not always better than any other. (Ideally, one would vary the padding type as a hyper-parameter I suppose.) I think intuitively that reflection and symmetric padding alters the local image structure and global statistics the least. (Clearly, the zeros from zero padding on the boundary are not truly part of the image, and the network has to learn this). However, these have different problems with "realism": for instance, reflection padding will "duplicate" a chair leg on the boundary if the padding is large enough, which may present its own problems depending on the task.
Overall, I think it doesn't matter too much, but it depends on the task and setup. For instance, in generative modelling and image-to-image translation, reflection padding avoid some artifacts on the boundary, as noted in the CycleGAN paper, for example.
An interesting recent paper is Partial Convolution based Padding by Liu et al, where they sidestep this issue by essentially having the convolution completely ignore the boundary. This is a partial convolution in the sense that it ignores the part of the filter that reaches outside of the image. The overall improvements seem to be relatively marginal, except that it does seem to help significantly on the boundary, as one would expect. See also this earlier work doing a similar thing in the segmentation context and this work which applies partial convolutions to inpainting. This approach to avoiding padding is seemingly well-principled.
TL;DR: test different paddings and see which is best for your architecture/task (also check the literature). If that's too much work, I default to reflect padding. If you want to get fancy or worry about boundary effects, use partial convolutions.
