Abstraction in Neural Networks

Question

In my nonlinear dynamics class in college, we discussed a simple perceptron with two input neurons and one output neuron that is trained on the patterns

pattern | Input | Output
   1    |  1,0  |   2
   2    |  3,1  |   6
   3    |  0,0  |   0

Solving the system of equations to determine the weights

$$w_1(1) + w_2(0) = 2$$ $$w_1(3) + w_2(1) = 6$$ $$w_1(0) + w_2(0) = 0$$

gives $w_1 = 2$ and $w_2 = 0$, which means the perceptron has made an abstraction. In more advanced neural networks, (a) how do you test for abstractions in your network and (b) how do you interpret what that abstraction means?

Answer 1

Since the weight of each feature (column) determines how important that column is in determining the output value, the abstraction means the most efficient and correct weights. For instance, here the features for pattern 1 are 1 and 0, and each of them are given a weight multiplying with which the output is calculated:

feature1 * weight 1 + feature2 * weight2 = output

So based upon the current pattern, the best values for weights to give the already extant outputs, are the ones given.

The test for the abstraction can be done via the compile method. First initial wights are assigned to features, then a single record or a patch of data passes through network and a cost function is calculated, then the error is backpropagated and new weights are set for the features (based on how much responsible each of them is for the error) and this continues.