What is the difference between Fully Connected layers and Bilinear layers in deep learning?


1 Answer 1


I quote the answers from What is a bilinear tensor layer (in contrast to a standard linear neural network layer) or how can I imagine it?.

A bilinear function is a function of two inputs $x$ and $y$ that is linear in each input separately. Simple bilinear functions on vectors are the dot product or the element-wise product.

Let $M$ be a matrix. The function $f(x,y)=x^TMy=\sum_iM_{ij}x_iy_j$ is bilinear in $x$ and $y$. In fact, any scalar bilinear function on two vectors takes this form. Note that a bilinear function is a linear combination of $x_iy_j$ whereas a linear function such as $g(x,y)=Ax+By$ can only have $x_i$ or $y_i$. For neural nets, that means a bilinear function allows for richer interactions between inputs.

Now what if you want a bilinear function that outputs a vector? Well, you simply define a matrix $M_k$ for each coordinate of the output and you end up with a stack of matrices. That stack of matrices is called a tensor (3-mode tensor to be exact). You can imagine the bilinear tensor product with two vectors as $x^⊀M_ky$ computed on each β€œslice” of the tensor.

Bilinear Models consists of two feature extractors whose outputs are multiplied using an outer product at each location of the image and pooled to obtain an image descriptor. 1

enter image description here

Its advantage is that it can model pairwise feature interactions in a translationally invariant manner, which is particularly useful for fine-grained categorization. It also allows end-to-end training using image labels only, and achieves state-of-the-art performance on fine-grained classification.

  • 1
    $\begingroup$ Great summary. Just one point -- for the Bilinear CNN model we directly took an outer product of x and y, which would mean an identity matrix for M. Since x and y are the result of learned projections anyway, one can assume anything that a matrix M would encode can be learned end-to-end in the features. $\endgroup$
    – AruniRC
    Commented Jan 31, 2019 at 18:33
  • $\begingroup$ I am stuck in the formula 𝑓(π‘₯,𝑦)=π‘₯𝑇𝑀𝑦=βˆ‘π‘–π‘€π‘–π‘—π‘₯𝑖𝑦𝑗. Can someone explain it? How do we multiply for instance $x^{T}A$. The dimensions don't seems to match. I am not sure how to do this matching actually since x is 2d and M is 3d. $\endgroup$ Commented Feb 10, 2021 at 6:50
  • $\begingroup$ For each image, they are done separately. You can search for vectorization. $\endgroup$ Commented Feb 10, 2021 at 7:01
  • $\begingroup$ @EduardoReis it is not simple matrix multiplication. It is done by einstein summation aka einsum. $\endgroup$
    – Ritwik
    Commented Jan 6, 2023 at 10:58
  • $\begingroup$ @GreenFalcon does bilinear functions works better than simple concatenation (of $x$ and $y$) and a linear layer? If yes, then why? $\endgroup$
    – Ritwik
    Commented Jan 6, 2023 at 11:00

Your Answer

By clicking β€œPost Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.