In the page 3 of the paper of EfficientNet, there is a equation $$\mathcal{N} = \bigodot_{i=1...s} \mathcal{F}_{i}^{L_i} \big(X_{\langle H_i, W_i, C_i \rangle}\big)$$ where $\mathcal{N}$ is the conv net and each $\mathcal{F}_i^{L_i}$ is the $i$th-stage layer operator that has length $L_i$.

What I don't understand is, what is this $\odot$ in this equation? Does the author refer to the Hadamard product or does he refer to the function composition? He previously mentioned that $\mathcal{N} = \mathcal{F}_k \odot ... \odot \mathcal{F}_1 (X_1)$, where k is the depth of the net. So I thought it means that $\odot$ is just function composition. But EfficientNet has skip connection. In the keras implementation it uses layers.merge.Multiply() so it can also means that the input data $X$ is multiplied with the transformed $\mathcal{F}(X)$, and $\odot$ maybe means the Hadamard product.

Does anyone knows the answer? Thanks.


It probably represents repeated function composition.

Key reasons:

  1. The authors' wording before this equation is: "a list of composed layers".
  2. A Hadamard product could not be applied to layers of different sizes, whereas function composition of course can.
  3. Yes, EfficientNet has skip connections, but I think they try to validate function composition via a distinction between "layers" and "stages". They say that "$F_i^{L_i}$ denotes layer $F_i$ is repeated $L_i$ times in stage $i$". They also explain that "all layers in each stage share the same architecture. Therefore, I think they put skip-connections in the form of function composition, but it is confusing. I think it's their way of compactly expressing the (often) repeated architecture of skip-connected layers.
  • 1
    $\begingroup$ Good points! Also when I typed the question I found that $\bigodot$ is rendered better in math mode than $\bigcirc$ (index is after the symbol but not below it), so it may also be the reason why they use this odot symbol to represent function composition. $\endgroup$ – Chris XU Sep 17 '20 at 0:58
  • $\begingroup$ @ChrisXU yea, they may have taken some notational liberties, idk. If you're ever in doubt, you can also email the corresponding author; they are usually happy to receive questions! In fact, for pre-prints, I think most would be appreciative of (constructive/courteous) feedback on clarity etc. for if/when they submit for peer review. $\endgroup$ – Benji Albert Sep 17 '20 at 1:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.