3
$\begingroup$

In a neural network, it is common to compute a dot product of the form

$$\langle w, x \rangle = w_1 x_1 + w_2 x_2 + \ldots + w_n x_n$$

and use it as argument to some activation function. This is done for several different weights $w$ and activation functions (one per neuron). The results are used as input to the next layer, where we compute again several dot products and use them as argument to more activation functions.

Using many layers leads do several theoretical difficulties of deep learning and one has to make some effort to make everything works. But at the end, hopefully it will end up working.

I was thinking in a way of "cheating" the use of many layers. The idea is to use just one layer in a way that the argument to the activation function is complex enough so the program will learn properly. Instead of a dot product I was thinking in using products of dot products:

$$\prod_{k=1}^L\langle w^{(i)}, x \rangle = (w_1^{(1)} x_1 + w_2^{(1)} x_2 + \ldots + w_n^{(1)} x_n) \cdot \ldots \cdot (w_1^{(L)} x_1 + w_2^{(L)} x_2 + \ldots + w_n^{(L)} x_n)$$

The result is them used as argument to some activation function. Since this argument already has several weights, my hope is that the need of several layers of activation functions wouldn't be necessary anymore. I even though about using sum of these products and tested it. I'm getting $\sim 80\%$ of accuracy in a classification problem where I know it is possible to get more than $90\%$. Does my approach has a fundamental limitation I'm not aware of?

Thanks.

Extra: Above I gave a simplified description of my idea (which I think is enough), but it may be helpful to put the whole thing here. My final model of learning is given by the following formula

$$ = \textbf{f}\left( \sum_{r=1}^R \left( \prod_{i=1}^{L-1} \langle \textbf{w}^{(i,r)}, \textbf{x} \rangle \right) \left[ \begin{array}{c} w_1^{(L,r)} \\ \vdots\\ w_m^{(L,r)} \\ \end{array} \right] \right) = $$

$$ = \left[ \begin{array}{c} f\left( \sum_{r=1}^R \sum_{i_1, \ldots, i_{L-1}=1}^n w_{i_1}^{(1,r)} \ldots w_{i_{L-1}}^{(L-1,r)} x_{i_1} \ldots x_{i_{L-1}} w_1^{(L,r)} \right) \\ \vdots\\ f\left( \sum_{r=1}^R \sum_{i_1, \ldots, i_{L-1}=1}^n w_{i_1}^{(1,r)} \ldots w_{i_{L-1}}^{(L-1,r)} x_{i_1} \ldots x_{i_{\ell-1}} w_m^{(L,r)} \right) \\ \end{array} \right]$$

$f:\mathbb{R} \to \mathbb{R}$ is the activation function and $\textbf{f} = (f, \ldots, f)$ is the "vectorized" version of this function. Note that I tried to come up with a very flexible formulation before the activation function. The idea is that this would avoid the need of more layers. I hope everything is clear at this point. Thanks again.

$\endgroup$
1
$\begingroup$

Well, you are basically just creating a big nth degree polynomial with no small degrees.

So yes, fundamentally flawed because you don't have a bias in your weights, and then the optimization will be very complex because each parameter/weight depends on all the others entries in the computation.

$\endgroup$
  • $\begingroup$ Thank for the answer! Well, I can try to impose some bias, this is doable. You think my model would work properly after that? $\endgroup$ – Integral Feb 13 at 18:09
  • 1
    $\begingroup$ It could, but the cost function will be far more complex to optimize. $\endgroup$ – Matthieu Brucher Feb 13 at 18:11
  • $\begingroup$ I was able to compute an explicit formula for the derivative of this thing. I can't say it has a low computational cost at the moment, but if $L$ and $R$ are low (like 2 or 3), the cost is acceptable. I'll try some variations before giving up. Do you some any suggestions? Thanks again! $\endgroup$ – Integral Feb 13 at 18:23
  • 1
    $\begingroup$ Oh yes, sorry, didn't notice the fact that $R$ was something else. But actually I don't know what it brings to the mix, as it's still a sum ont he same inputs, for the same outputs with the same equation, just different weights, leading to a large underdetermination of the $L$ weights. $\endgroup$ – Matthieu Brucher Feb 13 at 18:42
  • 1
    $\begingroup$ The $L$ issue is not a surprise, it's due tot he polynomial aspect of it. $\endgroup$ – Matthieu Brucher Feb 13 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.