Questions tagged [weight-initialization]
The weight-initialization tag has no usage guidance.
29
questions
1
vote
0answers
13 views
Can we talk about vanishing activations?
When updating the weights of a deep neural network using backpropagation, to update the weights of a given hidden layer, we use both the partial derivatives of the objective function with respect to ...
1
vote
1answer
32 views
Matrix factorization how to initialize weights and biases?
I have a matrix factorization and I'm wondering how I should initialize its weights and biases.
When getting prediction (recommendation), after computing a dot product and adding bias I want to use ...
3
votes
2answers
25 views
Result of uniform weight initialization in all neurons
Background
cs231n has the question regarding how to initialize weights.
Question
Please confirm or correct my understandings. I think the weight value will be the same in all the neurons with ReLU ...
2
votes
1answer
45 views
Why is it okay to set the bias vector up with zeros, and not the weight matrices?
We do not initialize weight matrices with zeros because the symmetry isn’t broken during the backward pass, and subsequently in the parameter updating process.
But it is safe to set the bias vector up ...
1
vote
0answers
15 views
Initializing weights that are a pointwise product of multiple variables
In two-layer perceptrons that slide across words of text, such as word2vec and fastText, hidden layer heights may be a product of two random variables such as positional embeddings and word embeddings ...
0
votes
1answer
24 views
Does training of neural networks follow the same order in each epoch?
Each epoch uses the weight from the end of the previous epoch(correct me if I am wrong). Is the updating of parameters after each batch always in the same order? To rephrase, are the batches always in ...
0
votes
1answer
22 views
Why don't different output weights break symmetry?
My deep learning lecturer told us that if a hidden node has identical input weights to another, then the weights will remain the same over the training/there will be no separation. This is confusing ...
1
vote
1answer
19 views
Weight-initialisation and model instability
Trying to calibrate a relatively vanilla NN, setting the hyper-parameter tuning aside*, it appears that weight initialisation has a lot of impact on the model output. Ie. Models calibrated with ...
0
votes
0answers
12 views
Neural network - calibration strategy & cross-validation
I have a hard time articulating the different parts of a calibration process of a relatively vanilla neural network. I am mostly concerned with :
Grid search for the regularisation hyperparameter
...
1
vote
2answers
38 views
What is wrong with a neural network model which is so dependent on the seed of initialization?
I have a fully-connected neural network with one hidden layer with 2 units which its goal is to classify a dataset with 324 samples and 300 features. 50% of the dataset is used for train and 50% of it ...
0
votes
0answers
33 views
Debugging VGG16 Convergence Issues: Effect of Image Normalization and Weight Initialization
I've implemented VGG16 in Keras with 2 output classes for the Kaggle 'dogs v. cats' dataset as a learning exercise. I'm trying to understand the training behavior that I'm seeing. For all my tests, I ...
0
votes
0answers
19 views
If input data to CNN is not normalised, how should I initialise the weights?
I've read that He normalisation is preferred for Relu activated CNN's. However, understanding how Relu's work by linearly activating positive inputs while zero or negative inputs are zero'd (with no ...
1
vote
0answers
37 views
Is the hyperbolic tangent function a solution to the weight clipping problem of WGAN?
Instead of clipping to the range $[-c,c]$ in WGAN (Wasserstein generative adversarial network), why not smoothly map into the range $[-c,c]$ by using $c\times \mathrm{tanh}(w)$? This would guarantee ...
1
vote
1answer
43 views
Should weight distribution change more when fine-tuning transformers-based classifier?
I'm using pre-trained DistilBERT model from Huggingface with custom classification head, which is almost the same as in the reference implementation:
...
4
votes
2answers
435 views
Compare Coefficients of Different Regression Models
in my project, I am using asuite of shallow and deep learning models in order to see which has the best performance on my data. However, in the pool of shallow machine learning models, I want to be ...
4
votes
1answer
659 views
Is it wrong to use Glorot Initialization with ReLu Activation?
I'm reading that keras' default initialization is glorot_uniform.
However, all of the tutorials I see are using relu ...
1
vote
0answers
42 views
How could we use a neuron instead of 100 neurons in 0.01 standard Weight initialization
Many books explain that we don't need to use 100 neurons and just can use a neuron if all result values(sigmoid(Wx)) are same at each of hidden layers.
but, I don't know that we can just use a ...
1
vote
0answers
43 views
Cold-start problem in Real Time Bidding
I'm currently on the reading stage of the deployment of an RTB system. I've seen the problem of a cold start (having no initial guess of how to bid) in several papers, but I haven't really seen it be ...
0
votes
1answer
24 views
Why is the variance going down so much in this weight initialization problem(using pytorch)?
first look at this example
>>> x = t.randn(512)
>>> w = t.randn(512, 500000)
>>> (x @ w).var()
tensor(513.9548)
it makes sense that ...
1
vote
1answer
18 views
Same probability for all classes
I implemented a fully connected MLP of shape [783 (input), 128 (hidden layer) and 10 (output)] the hidden layer had a sigmoid activation function and the output a sofmax.
I tested with the dataset of ...
1
vote
1answer
40 views
How can I have the same initialization for all my networks?
I want to have the same weights for layer initializations in all my networks, so that when I'm comparing their first epoch loss they all start from the same value. Is there a way in keras to do this?
...
2
votes
0answers
86 views
A model that only works by setting all initial weights to zero
In this model from MusicNet, they set the initial weights of their neural network to all zeros.
...
2
votes
3answers
336 views
weight training speed too slow in CNNs
I'm writing my own CNN code from scratch. Though I got fast, converged and satisfactory results, the trained weights change very little in value (while cost/loss function drops in time rapidly in a ...
3
votes
1answer
95 views
How large of a value should a weight have in a neural network?
If you're assigning random values to the weights in a neural network before back-propagation, is there a certain maximum or minimum value for each weight ( for example, 0 < w < 1000 ) or can ...
1
vote
1answer
107 views
Weights initialization in Neural Network
I was viewing code for custom neural network for sentiment analysis. It had 3 layers (1 hidden layer). I am more concerned with weight initialization for the layers
...
0
votes
2answers
168 views
C++ return array from function
I would like to implement machine learning algorithm in C++ without using any C++ machine learning library. So I'm writing this initializer function for generating zero matrices but can't figure out ...
4
votes
2answers
49 views
What are the reasons for drawing initial neural network weights from the Gaussian distribution?
Are there theoretical or empirical reasons for drawing initial weights of a multilayer perceptron from a Gaussian rather than from, say, a Cauchy distribution?
5
votes
2answers
8k views
what is difference between the DDQN and DQN?
I think I did not understand what is the difference between DQN and DDQN in implementation.
I understand that we change the traget network during the running of DDQN but I do not understand how it is ...
6
votes
2answers
5k views
What are the cases where it is fine to initialize all weights to zero
I've taken a few online courses in machine learning, and in general, the advice has been to choose random weights for a neural network to ensure that your neurons don't all learn the same thing, ...
