0
$\begingroup$

In the Cooperative stage of Kohonen's SOM, the neighborhood for a winning neuron(output node). In most cases, the neighborhood function happens to be the Gaussian Function. For example, $$h_j,_i = exp(-d_j,_i^2/2*\sigma^2)$$ where $h_j,_i$ is the window function ($i$ is the index of the winning neuron and $j$ is the index of the encompassing neuron), $d_j,_i$ is the lateral distance between the winning neuron $i$ and the excited neuron $j$.

Now the user who is going to implement the SOM has the input data and the randomly initialized weight vectors. How is $d_j,_i$ determined? Is it randomly initialized too? Also what is the relation between the weight vectors of the output nodes and their positions? The figure below is just to give a visual representation of the organization of the input and the output nodes

Just to give a visual representation of the organization of the input and the output nodes

$\endgroup$
0
$\begingroup$

The distance is calculated according to a distance function (euclidian, manhattan, mahalanobis and so on. As this is an unsupervised model (hence the self-organizing part in the name) there are no output neurons. Would you clarify the term?


I'll just recap the training a bit:

Inititalizing: We have a grid (let's assume 2d) of neurons $n_i = (w_i, k_i)$, where $w_i$ is a randomly initialized weight and $k_i$ the position on the grid.

  1. We'll now pick a training sample $x_i$ randomly. For this instance $x_i$ we pick the neuron $n_m$ ($m$ for minimum) where the distance between the weight vector and the instance is minimal given a distance function $d$, so $n_m = \text{argmin}_{n_j} \; d(x_j, W(n_j))$ where $W(\cdot)$ gives the weight for the respective neuron.
  2. We now pick a set of neurons for which we will adapt the weight vector given a neighbourhood function (for example your gaussian, or a cone). The weights that should have their weights updated are given by $N^{+t} = \{ n_i = (w_i,k_i) \mid d_A(k_m,k_i) \leq \delta^t \}$, where $t$ is the step in time (I'll mention that later), the "reach" $\delta^t$ the neighbourhood should have and of course the position on the grid $k_m$ for the winning neuron.
  3. We then update the weights of the neurons in the neighbourhood of the winning neuron $n_m$ according to some updating rule. This can be interpreted as moving the weights closer to the input we currently look at (because we chose the neuron with the closest weight vector). A possible update rule could be: $w_s^{t+1} = w_m^t + \epsilon^t \cdot h_{mi}^t \cdot (x_j - w_m^t)$, where $\epsilon^t$ is a time-dependent learning rate and $h_{si}^t$ weights the distance from the winning neuron to the neuron we are changing right now, also time-dependent.

In order to somewhat guarantee a topological mapping two things should be considered now:

  1. The learning rate, which is part of the update rule, has to be decreased over time, starting with a relatively high value.
  2. The neighbourhood "radius" has to be decreased as well, also starting from a relatively high value.

Let's visualize this on this picture real quick. The red nodes are inputs. The dark grey is the winning neuron for the input that is considered right now (the picture does not make it clear that we look at one example/input at a time!). The light grey nodes are the ones that are within the neighbourhood of the winning neuron and hence are updated slightly according to some updating rule.

enter image description here


Disclaimer: this was from the top of my head right now, please check any formulas or hypotheses I propose here before using it in any kind of work. (Which you should always do!) . See also wikipedia, they also have some explanation of the algorithm that might better suit you.

Image source Additioanl source (German Wikipedia)

| improve this answer | |
$\endgroup$
  • $\begingroup$ In the second step, you pick a set of neurons. Those are the nodes/neurons I'm talking about, the output nodes. How do you pick those neurons? Picking includes specifying their position in let's say a 2-D plane. $\endgroup$ – Viswanath Hariharan Apr 16 '17 at 14:08
  • $\begingroup$ Also, in the third step, you say that updating the weights can be interpreted as the weights of the neurons moving closer to the input we are currently looking at. So they aren't moving towards the input actually? If they aren't, what is actually happening? How is the topology of the input retained if the weights aren't actually moved? $\endgroup$ – Viswanath Hariharan Apr 16 '17 at 14:10
  • $\begingroup$ I updated the explanation with respect to the first comment. Regarding your second comment: Of course they are moving towards the input. We choose the neuron with the most similar weight with the least distance to the input. You can also say you choose the weight most similar to the input. Why wouldn't we then move the weights of the neighbouring neuron weights towards the input? $\endgroup$ – C. S. Apr 16 '17 at 17:38
  • $\begingroup$ Okay, so we are moving the weights of the neurons towards the input. This is accomplished by updating the weights of the neurons right? So after moving the neuron, its position changes. By how much and what decides this? If you say updating the weight vectors decides this, what is the relation between the position of the neurons or the distance moved by the neurons and their respective weights? $\endgroup$ – Viswanath Hariharan Apr 16 '17 at 21:39
  • $\begingroup$ The neurons don't move. We just change the weights. They stay fixed on the very position of the frame we initialized them on. Could you rephrase your question? $\endgroup$ – C. S. Apr 16 '17 at 21:56
0
$\begingroup$

I am struggling with the same question. I think the most import answer on this is this comment of C. S.:

The neurons don't move. We just change the weights. They stay fixed on the very position of the frame we initialized them on.

In my understanding the positions aren't changed at all. The only propose of the grid is to define the neighborhood of each node. Neither the neighborhood, nor the distances of the map space change. Everything dynamic happens in the weight space.

Images like this are very confusing, though: enter image description here

But if you closely read the description, it says, it displays the distance in the weight space.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.