I understand that from mathematical point of view, only difference between Convolution and Cross-Correlation is that Convolution is commutative, while Cross-Correlation is now.

In many articles Cross-Correlation is explained as measuring similarity between two signals, and Convolution is explained as calculating relationship between two signals.

My question is this, what is the point of commutative property? How does it make Convolution different from Cross-Correlation in practice? Or is it just there for "better" mathematics and easier proofs? Also, what are, if any, benefits of implementing CNN with Convolution over Cross-Correlation?


Indeed, convolution and cross-correlation are closely related. The former is a bit more natural in some areas of mathematics; most notably, in the convolution theorem for the Fourier transform, which states that the Fourier transform of the convolution of two functions is equal, under certain conditions, to the product of their Fourier transforms: $$ \mathcal{F}\{f * g\} = \mathcal{F}\{f\} \mathcal{F}\{g\}, $$ thus providing an efficient way to compute the convolution.

In the context of CNN, there difference between the convolution and the cross-correlation is irrelevant. In the discrete two-dimensional case $$ (f {**} g)[n_1, n_2] = \sum_{m_1 = -\infty}^{+\infty}\sum_{m_2 = -\infty}^{+\infty} f[m_1, m_2] \, g[n_1 - m_1, n_2 - m_2], $$ $$ (f' {\star\!\star} g)[n_1, n_2] = \sum_{m_1 = -\infty}^{+\infty}\sum_{m_2 = -\infty}^{+\infty} \overline{f'[m_1, m_2]} \, g[n_1 + m_1, n_2 + m_2]. $$ If $f$ is the filter and $g$ is the output of the previous layer of the CNN, you can see that the convolution is equivalent to cross-correlation with a filter $$ f'[m_1, m_2] = \overline{f[-m_1, -m_2]}, $$ which is just a reflection around the secondary diagonal since all values are real.

In short, there are no benefits to using the true convolution in CNN. The only difference would be that the learned filters would be mirrored about the secondary diagonal.

| improve this answer | |
  • $\begingroup$ Awesome! Thanks for the great answer! $\endgroup$ – Stefan Radonjic Mar 15 at 21:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.