# How exactly does adding a new unit work in Cascade Correlation?

I've just read The Cascade-Correlation Learning Architecture by Scott E. Fahlman and Christian Lebiere.

I think I've got the overall concept (or at least the "cascade" part - a 4min YouTube video how I think it works):

1. Start with a minimal network with input and output units only
2. Learn those weights with standard algorithms (e.g. gradient descent - they seem to use another training objective which I don't quite understand, so it is gradient ascent in the paper)
3. When the network doesn't improve, add a single new hidden unit. This unit gets input from all input nodes and all hidden nodes which were added before. Its output goes to all output nodes only.
4. Repeat step 3

However, I don't understand the details of step 3: The input weights to hidden units are frozen (indicated by boxes in the paper). When exactly do they get frozen? Are they just initialized by random and never learned at all?

I also don't understand this paragraph:

To create a new hidden unit, we begin with a candidate unit that receives trainable input connections from all of the network's external inputs and from all pre-existing hidden units. The output of this candidate unit is not yet connected to the active network. We run a number of passes over the examples of the training set, adjusting the candidate unit's input weights after each pass. The goal of this adjustment is to maximize $S$, the sum over all output units $o$ of the magnitude of the correlation (or, more precisely, the covariance) between $V$, the candidate unit's value, and Eo, the residual output error observed at unit o. We define S as

$$S = \sum_{o} | \sum_p (V_p - \bar V) (E_{p,o} - \bar{E_o}) |$$

where $o$ is the network output at which the error is measured and p is the training pattern. The quantities $\bar V$ and $\bar{E_o}$ are the values of $V$ and $E_o$ averaged over all patterns.

What is an "residual output error"? Is $V_p$ simply the activation of the unit given the pattern $p$? What does the term $S$ mean and why do we want to maximize it?

• Just wanted to tell you: "I'm a huge fan of your questions" :) – Dawny33 Jan 7 '16 at 10:28

• We then use that equation $$S=\sum_o \left | \sum_p(V_p - \overline{V})(E_{p,o} - \overline{E_o}) \right |$$ to train the candidate nodes.
• $S$ is the correlation between the activation of our candidate node and the the residual error.
• $V_p$ is the activation of the candidate node given input $p$.
• After a bit of backprop training against $S$ we choose our best candidate. This becomes a new hidden node.