So, I am new to using Tensorflow, and I am trying to create a neutral network, and in all the guides I saw one of the below lines of code was used.
Weight1 = tf.Variable(tf.truncated_normal([-,-]))
or...
Weight1 = tf.Variable(tf.random_uniform([-,-]))
Both of them work in the examples, but I want to know is what is the difference between them and is one better for certain situations than the other?
A simple Google search would show you the differences.
Truncated Normal outputs random values from a truncated normal distribution. The generated values follow a normal distribution with specified mean and standard deviation, except that values whose magnitude is more than 2 standard deviations from the mean are dropped and re-picked.
Random Uniform outputs random values from a uniform distribution. The generated values follow a uniform distribution in the range [minval, maxval). The lower bound minval is included in the range, while the upper bound maxval is excluded.
Regarding when is it better to use one or the other, we don't know. Weights just need to be random, and there is no conclusive research that shows that one or another distribution is better for specific scenarios. That being said, there are better ways to initialize weights that completely randomly. These include the very popular Xavier initializer.
The basic difference between tf.truncated_normal and tf.random_normal is the former one generate truncated values following a normal distribution with specified mean and standard deviation, whereas the later one outputs random values from a normal distribution.
Why to use truncated values ?
The truncated normal distribution is used to prevent generating dead neurons.
One should generally initialize weights with a small amount of noise for symmetry breaking, and to prevent 0 gradients. Since we're using ReLU neurons, it is also good practice to initialize them with a slightly positive initial bias to avoid dead neurons.
Here it has been well answered.
