Step-by-step construction of an RBF neural network

Question

I have started to look at Radial-Basis-Function Neural Networks (RBF-NN), and would like to solve the following task as an exercise: Given is the data in the image below. Each output $Y_j$ of the network is defined as:

$(*)$ $Y_j = \sum_{i}^N w_{ij}* exp(-\frac{||x-\mu_i||^2}{2\sigma_i^2})$

for the -ith neuron. The task is to draw an RBF network that perfectly classifies the data, with suitable means, covariances and weights. In a second step a point has to be taken and classified in the worked-out model.

My ideas We have two nodes in the input layer, one for each dimension. The hidden-layer has as many neurons as there are training samples. Each of these calculates the activation given by the exponential above. The output layer has three output nodes as there are three classes. I would determine $\Sigma$ and $\mu$ as follows: For each class and training-sample $x_i$ of this class, $\mu_j$ is just the centroid of the training samples, $\sigma = \frac{1}{m}\sum_i ||x_i-\mu||$, and $\Sigma = \sigma*I_d$.

Questions That does not seem to be correct however - if I have the same $\Sigma$ and $\mu$ for all the hidden nodes of the same class, each new test-input would result in the same activation for all these nodes..so what exactly are the particular $u_i$ and $\sigma_i$? Also, for perfect classification, I would set all the weights which belong to the correct class to 1, and all the others to zero - would that make sense?

Answer 1

The hidden-layer has as many neurons as there are training samples.

This already sounds a bit fishy to me. The hidden layer of an RBF network resembles the Radial Basis functions. Thus, you have to think about how many centers shall your network have. This should not correspond to the number of training samples your have, since this would force the network to learn by heart. Let's look at the example you gave in the question:

Here I propose to use a network with four hidden neurons, corresponding to four RBF centers, marked with stars. This is, in the end, how your network should learn to generalize over the data you feed into it. Note that you do not have to set the class of an RBF network in prior, the network is supposed to learn the classes of each neuron by itself. How this is done is properly explained on the wikipedia page. A nice thing about RBF networks is that there is an analytical solution (over the Moore-Penrose pseudoinverse) for your weight vector.

That does not seem to be correct however - if I have the same $\Sigma$ and $\mu$ for all the hidden nodes of the same class, each new test-input would result in the same activation for all these nodes..so what exactly are the particular $u_i$ and $σ_i$?

I suppose you are talking about the centers $\mu_i$ and their width $\sigma_i$. These values, like described in the link above, can be set via a clustering over the data you have. I've also seen applications that adapt the centers online, but this is usually not necessary. A simple unsupervised learning / clustering approach is typically sufficient if you have all training data available.

Also, for perfect classification, I would set all the weights which belong to the correct class to 1, and all the others to zero - would that make sense?

If your designed network is able to solve your task, the analytical solution will do the job. Wikipedia describes how it is calculated.