I see a lot of sites talk that we can substitute 1x3 conv + 3x1 conv for 3x3 conv.
In order to demonstrate easily, we use a 3x3 image as an example.
From the point of view of parameters, I know that we will get fewer parameters and end up with the same result.
If the image(3x3) perform a 3x3 conv, we will get a 1x1 scalar, and this kernel size is 3x3, means we have 9 parameters.
On the other hand, if the image first performs a 1x3 convolution, and then performs a 3x1 convolution on its output. It will intuitively get the same result, a 1x1 scalar, and this kernel size is 2x3x1 = 6, which means we have 6 parameters.
However, from the point of view of calculation time, I don't think 3x3 should be substituted by 1x3 & 3x1.
For example, if we use the image to perform a 3x3 kernel, we should do 9 times of pixel-wise multiply operations, and then one time of addition. Totally 10 operations to get the result.
But, if we use the image(3x3) to perform a 3x1 kernel, we have to implement [3(multiply) + 1(add)] * 3 times to get a 1x3 feature map, then we still need to perform a 1x3 kernel. It will then takes 3(multiply) + 1(add) operations. Totally 16 operations throughout the whole process.
What I want to show is that although we reduce the parameters from replacing 3x3 conv to 3x1 & 1x3 conv, we still need more time to calculate during the training process.
Training spends lots of time and implements 3x1 & 1x3 doesn't save our time. So, should we substitute 1x3 conv + 3x1 conv for 3x3 conv?