1
$\begingroup$

I was viewing code for custom neural network for sentiment analysis. It had 3 layers (1 hidden layer). I am more concerned with weight initialization for the layers

 self.weights_0_1 = np.zeros((self.input_nodes,self.hidden_nodes))
 self.weights_1_2 = np.random.normal(0.0, self.output_nodes**-0.5,(self.hidden_nodes,self.output_nodes))

What is the idea behind initializing zero's weight matrix. I have learned that initializing weights to zero might lead to linearity.

This might be a very vague question, I will be happy to provide any specifics you want. https://github.com/udacity/deep-learning/blob/master/sentiment-network/Sentiment_Classification_Solutions.ipynb

$\endgroup$
  • 1
    $\begingroup$ Please provide the source, because as you already pointed out this is generally a bad idea. $\endgroup$ – feliks Dec 23 '18 at 11:07
1
$\begingroup$

Three layers with one hidden layer? This sounds wrong. You have 2 layers, input -> hides -> output, that's only 2 layers, with two sets of weights.

A single layer that gets initialized with only 0 will not converge, because the derivatives will be strictly identical (this rule also applies for any such last layer in a network, the layer gets basically useless). Here, there is a second layer after that saves the day, but it's still a bad practice in general.

$\endgroup$
  • $\begingroup$ I was considering Input, Output and hidden layer as three layer. Anyway I have edited the question, glad if you can help. $\endgroup$ – Sagar Dhungel Dec 23 '18 at 21:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.