How exactly does adding a new unit work in Cascade Correlation?

Question

I've just read The Cascade-Correlation Learning Architecture by Scott E. Fahlman and Christian Lebiere.

I think I've got the overall concept (or at least the "cascade" part - a 4min YouTube video how I think it works):

1. Start with a minimal network with input and output units only
2. Learn those weights with standard algorithms (e.g. gradient descent - they seem to use another training objective which I don't quite understand, so it is gradient ascent in the paper)
3. When the network doesn't improve, add a single new hidden unit. This unit gets input from all input nodes and all hidden nodes which were added before. Its output goes to all output nodes only.
4. Repeat step 3

However, I don't understand the details of step 3: The input weights to hidden units are frozen (indicated by boxes in the paper). When exactly do they get frozen? Are they just initialized by random and never learned at all?

I also don't understand this paragraph:

To create a new hidden unit, we begin with a candidate unit that receives trainable input connections from all of the network's external inputs and from all pre-existing hidden units. The output of this candidate unit is not yet connected to the active network. We run a number of passes over the examples of the training set, adjusting the candidate unit's input weights after each pass. The goal of this adjustment is to maximize $S$, the sum over all output units $o$ of the magnitude of the correlation (or, more precisely, the covariance) between $V$, the candidate unit's value, and Eo, the residual output error observed at unit o. We define S as

$$S = \sum_{o} | \sum_p (V_p - \bar V) (E_{p,o} - \bar{E_o}) |$$

where $o$ is the network output at which the error is measured and p is the training pattern. The quantities $\bar V$ and $\bar{E_o}$ are the values of $V$ and $E_o$ averaged over all patterns.

What is an "residual output error"? Is $V_p$ simply the activation of the unit given the pattern $p$? What does the term $S$ mean and why do we want to maximize it?

Answer 1

I've been reading up on cascade correlation quite a bit recently and made a python implementation https://github.com/DanielSlater/CascadeCorrelation (though it still needs a bit of cleaning up/extra work and has a bunch of me mucking around with using Particle Swarm Optimization for selecting candidates, definitely not production ready).

To try and explain step 3.

• Start by creating a number of candidate hidden nodes with random weights. These have incoming connection from all existing hidden nodes and input nodes.
• We then use that equation $$S=\sum_o \left | \sum_p(V_p - \overline{V})(E_{p,o} - \overline{E_o}) \right |$$ to train the candidate nodes.
• The residual output error is the difference between the output of the network and the target value(think sum of squared error without the square).
• $S$ is the correlation between the activation of our candidate node and the the residual error.
• $V_p$ is the activation of the candidate node given input $p$.
• After a bit of backprop training against $S$ we choose our best candidate. This becomes a new hidden node.
• This is when it's weights are frozen. That is to say after random initialization then back-prop training, then selecting the best one.