Questions tagged [weight-initialization]

Use this tag when asking about the weight initialization of neural networks which are used in machine and deep learning.

21 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
1answer
1k views

Is it wrong to use Glorot Initialization with ReLu Activation?

I'm reading that keras' default initialization is glorot_uniform. However, all of the tutorials I see are using relu ...
4
votes
2answers
92 views

What are the reasons for drawing initial neural network weights from the Gaussian distribution?

Are there theoretical or empirical reasons for drawing initial weights of a multilayer perceptron from a Gaussian rather than from, say, a Cauchy distribution?
2
votes
0answers
25 views

Where Does the Normal Glorot Initialization Come from?

The famous Glorot initialization is described first in the paper Understanding the difficulty of training deep feedforward neural networks. In this paper, they derive the following uniform ...
2
votes
0answers
210 views

A model that only works by setting all initial weights to zero

In this model from MusicNet, they set the initial weights of their neural network to all zeros. ...
1
vote
0answers
17 views

Is saddle point a cause for the vanishing gradient problem

I am a beginner to neural networks and I am writing a report summarising on the causes and solutions to the vanishing gradient problem. From what I have read, the 2 main causes are the repeated ...
1
vote
0answers
26 views

Question regarding weight initialization of an artificial neural network

This is what i'm trying to implement in Python. w0,...,w8 = vector w1 of shape (9,1) w9,...,w11 = vector w2 of shape (3,1) b0 (first bias) is of shape (3,1) b1 is of shape (1,1) vector X is of shape (...
1
vote
0answers
246 views

Bad marshal data YoLo model

I tried to run a project from repo and got the following log which, I believe, tells a problem with weights load. ...
1
vote
0answers
13 views

Can we talk about vanishing activations?

When updating the weights of a deep neural network using backpropagation, to update the weights of a given hidden layer, we use both the partial derivatives of the objective function with respect to ...
1
vote
1answer
51 views

Matrix factorization how to initialize weights and biases?

I have a matrix factorization and I'm wondering how I should initialize its weights and biases. When getting prediction (recommendation), after computing a dot product and adding bias I want to use ...
1
vote
0answers
33 views

Initializing weights that are a pointwise product of multiple variables

In two-layer perceptrons that slide across words of text, such as word2vec and fastText, hidden layer heights may be a product of two random variables such as positional embeddings and word embeddings ...
1
vote
0answers
60 views

Is the hyperbolic tangent function a solution to the weight clipping problem of WGAN?

Instead of clipping to the range $[-c,c]$ in WGAN (Wasserstein generative adversarial network), why not smoothly map into the range $[-c,c]$ by using $c\times \mathrm{tanh}(w)$? This would guarantee ...
1
vote
1answer
88 views

Should weight distribution change more when fine-tuning transformers-based classifier?

I'm using pre-trained DistilBERT model from Huggingface with custom classification head, which is almost the same as in the reference implementation: ...
1
vote
0answers
42 views

How could we use a neuron instead of 100 neurons in 0.01 standard Weight initialization

Many books explain that we don't need to use 100 neurons and just can use a neuron if all result values(sigmoid(Wx)) are same at each of hidden layers. but, I don't know that we can just use a ...
1
vote
0answers
51 views

Cold-start problem in Real Time Bidding

I'm currently on the reading stage of the deployment of an RTB system. I've seen the problem of a cold start (having no initial guess of how to bid) in several papers, but I haven't really seen it be ...
1
vote
1answer
23 views

Weight-initialisation and model instability

Trying to calibrate a relatively vanilla NN, setting the hyper-parameter tuning aside*, it appears that weight initialisation has a lot of impact on the model output. Ie. Models calibrated with ...
0
votes
0answers
42 views

Keras model's embedding weight get NaN value

I am working on 3 categorical and 19 numerical features in which I plan to use trained embedding weights (from categorical features). After training, and get weights from embedding layers, I got NaN ...
0
votes
0answers
16 views

Create weights network with randomly initialized weights for Keras Models

I work with a tool for audio feature extraction which has several layers (DenseNet, etc) for the extraction. The default is to use pre-trained imagenet weights, however I want to evaluate the ...
0
votes
0answers
12 views

Why checkpoint loss is different?

I am training a Mask RCNN model in Keras. I used checkpoints to save weights so I can resume training with the last optimized values. However, the loss is different when I save the checkpoint and ...
0
votes
0answers
51 views

Sklearn compute_class_weight function leads to inaccurate result

I am trying to fit CNN to imbalanced data with 3 classes and therefore I am using compute_class_weight function from SKLearn. The code is presented below. ...
0
votes
1answer
34 views

Update of mean and variance of weights

I'm trying to understand the Bayes by Backprop algorithm from the paper Weight Uncertainty in Neural Networks, the idea is to make a NN in which each weight has it's own probability distribution. I ...
0
votes
0answers
15 views

Gradients vanishing despite using Kaiming initialization

I was implementing a conv block in pytorch with activation function(prelu). I used Kaiming initilization to initialize all my weights and set all the bias to zero. However as I tested these blocks (by ...