What are the pros and cons of zero padding in a convolution layer?

Question

TensorFlow's conv2d() operation lets you choose between "VALID" (without padding) and "SAME" (with zero-padding). I suppose all other frameworks let you do the same. I'm trying to understand the pros and cons of zero padding: when would you want to use it, or not to use it?

So far, my understanding is that if the filter size is large relative to the input image size, then without zero padding the output image will be much smaller, and after a few layers you will be left with just a few pixels. So to maintain a reasonably sized output, you need zero-padding + stride 1. Is this the main reason for using zero-padding? Is it preferable to avoid it when you can, for example when the filter size is small relative to the input image size?

Answer 1

I summarized from some other web pages:

Pros: Deep network would be possible because the output dimension could be constant after convolution
     Example: Saving image border information

Cons: Heavy computation
     Example: Waste of computational resources