# How sensitive are feedforward Neural Network to weight changes?

Let's consider the space of feedforward neural networks with a given structure: $$L$$ layers, $$m$$ neurons per layer, ReLu activation, input dimension $$d$$, output dimension $$k$$.

Which means I'm considering the map $$F: \mathcal{W}_1 \times \mathcal{W}_2 \times \dots \times \mathcal{W}_L \times \mathbb{R}^d \to \mathbb{R}^k$$, where $$\mathcal{W}_i$$ is the space of possible weights for layer $$i$$. We also assume, for simplicity, that every weight matrix has a norm upper bounded by a constant $$M$$. Let's now assume that I have fixed parameters so that we obtain $$v = F(W_1, \dots, W_L, x^*) \in \mathbb{R}^k$$ (note that $$x^*$$ is fixed as well).

Now imagine that I inject some random noise $$\eta \in \mathbb{R}^{m \times m}$$ in a weight matrix $$W_i$$, where the norm of the noise is 10% of the norm of the matrix, e.g. $$||\eta|| = ||W_i||/10$$ . How does it affect my final output?

which means, what's the expected value of $$||v - v_*||$$, where $$v_*$$ is the output of the network obtained after the small changes in the weights described before?

Note that this has nothing to do with the learning process, it's just about the sensitivity/resistance of a neural network with respect to random noise injected in a weight matrix.

• I'm not talking about the learning process, I'm assuming that the parameters of the neural network are fixed. I'm looking at the NN as a mathematical function which takes as an input points in a $d$-dimensional vector space and outputs points in a $k$ dimensional one. Now, once I have this function, which has fixed parameters, I add the noise mentioned above. My question is how this noise affects the output of the function. I'll edit because I think I wasn't clear enough – Alfred Jan 13 '19 at 19:49