I have a conceptual problem regarding the overall learning capability of a neural network differentiated by the different types of input that we can give to the network. Suppose that we have a feedforward neural network. This network has only one output. We can give several different kinds of inputs, $I_k$, to the network in which $k$ denotes the different kinds of inputs that we have. Just to be clear, by input I don't mean the nodes which come before the layer containing $x_i$ but rather a function which the network receives as an input. to be more clear, suppose that each $x_i$ is a function of $I_k$.

enter image description here

We know that in order for the network to learn, backpropagation algorithm should work efficiently. Suppose that we fix the output $y=y_{0}$. We calculate the possible values for $x_i$ which can produce the output $y_0$ using the input $I_1$ first and the using another type of input $I_2$. Now we increase the desired output which we use to construct possible $x_i$ values by $\Delta y$ so that $y_1=y_0 + \Delta y$ and do the same calculations again. The idea that I have in my mind is as follows:

In order for the packpropagation to work efficiently, any change for the desired output $y_1 - y_0 = \Delta y$ should be reflected to the previous layer. Assume that we have a large number of $x_i$ (not just 5). We find $x_i^{(1)}$ and $x_i^{(2)}$, corresponding to the output $y_1$ and $y_2$ respectively and substract their norms:


in which $\bar{x}$ denotes the vector of $x_i$. Can we say that if for the input $I_1$ the value for $\Delta\bar{x}$ is bigger than that for $I_2$, then the network has a better learning capacity using the input $I_1$ than the $I_2$? Basically the idea is to find an overall reflection of the changes of the output in the previous layer and use it as a criteria to differentiate between the learning capabilities of the networks. Any comment is highly appreciated.

  • $\begingroup$ If the picture you used doesn't describe your problem you should use a different picture. It sounds like you have y(x(I)) and you want to find dI/dy. Are you defining learning capacity as \(\Delta\bar{x}\)? $\endgroup$
    – Derek H
    Sep 26 at 13:09
  • $\begingroup$ I have updated the question if you are interested. All I want is a criteria which can decide for what input I can train the network more efficiently. $\endgroup$
    – Ali Pedram
    Sep 26 at 13:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.