1
$\begingroup$

I see a lot of sites talk that we can substitute 1x3 conv + 3x1 conv for 3x3 conv.

In order to demonstrate easily, we use a 3x3 image as an example.

From the point of view of parameters, I know that we will get fewer parameters and end up with the same result.
If the image(3x3) perform a 3x3 conv, we will get a 1x1 scalar, and this kernel size is 3x3, means we have 9 parameters.
On the other hand, if the image first performs a 1x3 convolution, and then performs a 3x1 convolution on its output. It will intuitively get the same result, a 1x1 scalar, and this kernel size is 2x3x1 = 6, which means we have 6 parameters.

However, from the point of view of calculation time, I don't think 3x3 should be substituted by 1x3 & 3x1.

For example, if we use the image to perform a 3x3 kernel, we should do 9 times of pixel-wise multiply operations, and then one time of addition. Totally 10 operations to get the result.

But, if we use the image(3x3) to perform a 3x1 kernel, we have to implement [3(multiply) + 1(add)] * 3 times to get a 1x3 feature map, then we still need to perform a 1x3 kernel. It will then takes 3(multiply) + 1(add) operations. Totally 16 operations throughout the whole process.

What I want to show is that although we reduce the parameters from replacing 3x3 conv to 3x1 & 1x3 conv, we still need more time to calculate during the training process.

Training spends lots of time and implements 3x1 & 1x3 doesn't save our time. So, should we substitute 1x3 conv + 3x1 conv for 3x3 conv?

$\endgroup$
0
$\begingroup$

A 3x3 convolution can do things a 1x3 followed by a 3x1 cannot do, for instance diagonal filtering.

As you said, it's all about the number of parameters, the first convolution has far more capabilities than the second solution.

$\endgroup$
  • $\begingroup$ In that case, why Inception convnet use it so often, there must be some reason. $\endgroup$ – Evans Tsai Dec 20 '18 at 9:05
  • $\begingroup$ If you don't need the diagonal filtering for some reason, then no need to estimate a 3x3 convolution. This is all about network design and what question you are asking the network and what question you want to get answered with what data. $\endgroup$ – Matthieu Brucher Dec 20 '18 at 9:06
  • $\begingroup$ @MatthieuBrucher could you give an example in which case diagnonal filtering is usefull ? $\endgroup$ – deltascience Mar 31 at 9:15
  • $\begingroup$ Any meaningful filter?? Look at any filter generated by a CNN. $\endgroup$ – Matthieu Brucher Mar 31 at 10:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.