Skip to main content

Questions tagged [weight-initialization]

Use this tag when asking about the weight initialization of neural networks which are used in machine and deep learning.

Filter by
Sorted by
Tagged with
0 votes
0 answers
26 views

Intuition Behind Xavier Initialization

I am trying to understand the intuition behind why xavier works. So far I have pieced togther that $Var(Z)=n_{in}*Var(X)*Var(W)$ and so if we want the variance to mitigate the variance diminishing ...
Stef's user avatar
  • 1
0 votes
0 answers
22 views

Clarifying the arguments of "Understanding the difficulty of training deep feedforward neural networks"

I made the decision to try to push through the paper "Understanding the difficulty of training deep feedforward neural networks". (The paper is given as a reference in "Hands-On Machine ...
Chris's user avatar
  • 101
0 votes
1 answer
44 views

Why the standard deviation of the BERT weight initialization is 0.02 by default

The purpose of weight initialization in the neural network is to keep the variance of calculation output in the layers to 1.0, and it depends on the calculations involved in the layers. Initializing ...
mon's user avatar
  • 711
0 votes
1 answer
48 views

Train CNN weights by using FFT - Reinforcement Learning?

Assume that you are doing convolution inside a CNN network, by using FFT because FFT is much way faster than using 4-5 for-loops etc. But how should I train the weights if I know the output of my CNN ...
euraad's user avatar
  • 115
0 votes
0 answers
21 views

How to put WEIGHT to the binary data?

I have a dataset with ship and vessel AIS messages as well as other pertinent information. Names of dataset features are similar to those in the table below. Data from the Hamelin and Minden locks ...
Firoj Ahmmed Patwary's user avatar
1 vote
0 answers
15 views

confusion with Xavier Initiliazation definition

When researching online, I keep finding that Xavier/Glorot initialization is: however, the original paper by Glorot said that this was a common initialization strategy that they soon found did not ...
tom394's user avatar
  • 11
0 votes
0 answers
58 views

CNN sharing weights in feature map

what do they mean when they say all neurons in a channel share weights with one another? Do they mean that in a chanel or a featue map the weights are the same ?
heyoka955's user avatar
2 votes
0 answers
92 views

TF: What is the difference between the 'kernel weights' and the 'recurrent kernel weights' in LSTMs/GRUs?

Context: I am trying to understand the differences between the GRU/LSTM cells from tensorflow and pytorch (for research reproducibility) and noticed that TensorFlow differentiates between the ...
Robin van Hoorn's user avatar
0 votes
0 answers
97 views

Effect of torch weight_norm when dim = None

torch.nn.utils.weight_norm(module, name='weight', dim=0) When dim = None, g parameter becomes equal to $\|v\|$. Therefore, $w=g \frac{v}{\|v\|} = v$. So, I think ...
alryosha's user avatar
  • 101
1 vote
1 answer
155 views

How to add significance weighting in user based collaborative filtering

I have been learning about recommender systems these past days. More specifically about the collaborative filtering. While exploring I found that it can be useful to use "significance weighting&...
ilved17's user avatar
  • 31
0 votes
1 answer
421 views

pytorchs LSTMs use of 'bias' and 'weight' strings

Hi I am new to RNN and have come across this the following implementation of Pytorchs LSTM, but I cant understand how (or why) the 'bias' and ...
Piskator's user avatar
  • 125
0 votes
2 answers
136 views

NNs for fitting highly oscillatory functions

in a scientific computing application of neural networks, I have to maximize several neural networks with scalar output with respect to a target/loss function (coming from a weak form of a PDE). It is ...
PM25's user avatar
  • 1
4 votes
4 answers
2k views

Why are deep learning models unstable compare to machine learning models?

I would like to understand why deep learning models are so unstable. Suppose I use the same dataset to train a machine learning model multiple times (for example logistic regression) and a deep ...
Kyv's user avatar
  • 151
1 vote
0 answers
77 views

Is saddle point a cause for the vanishing gradient problem

I am a beginner to neural networks and I am writing a report summarising on the causes and solutions to the vanishing gradient problem. From what I have read, the 2 main causes are the repeated ...
siegfried's user avatar
  • 113
6 votes
1 answer
3k views

Where Does the Normal Glorot Initialization Come from?

The famous Glorot initialization is described first in the paper Understanding the difficulty of training deep feedforward neural networks. In this paper, they derive the following uniform ...
Hermi's user avatar
  • 167
0 votes
0 answers
48 views

Create weights network with randomly initialized weights for Keras Models

I work with a tool for audio feature extraction which has several layers (DenseNet, etc) for the extraction. The default is to use pre-trained imagenet weights, however I want to evaluate the ...
Link's user avatar
  • 21
1 vote
0 answers
60 views

Question regarding weight initialization of an artificial neural network

This is what i'm trying to implement in Python. w0,...,w8 = vector w1 of shape (9,1) w9,...,w11 = vector w2 of shape (3,1) b0 (first bias) is of shape (3,1) b1 is of shape (1,1) vector X is of shape (...
user's user avatar
  • 11
1 vote
1 answer
74 views

Shared classifier for 3 neural networks (is this weights sharing?)

I would like to create 3 different VGGs with a shared classifier. Basically, each of these architectures has only the convolutions, and then I combine all the nets, with a classifier. For a better ...
CasellaJr's user avatar
  • 229
1 vote
0 answers
1k views

Bad marshal data YoLo model

I tried to run a project from repo and got the following log which, I believe, tells a problem with weights load. ...
Michael's user avatar
  • 131
0 votes
1 answer
59 views

CNN Design for Counting on Simple Images

This is the first CNN I'm designing following college examples and assignments. I'm working on a CNN that I'd like to use to classify images by the number of shapes on them. My basic problem is that I ...
AndyC's user avatar
  • 3
1 vote
1 answer
109 views

How to force a NN to ouput the same output given a reverse input?

I want to choose an architecture that can deal with an input symmetry. As input, I have a sequence of zeros and ones, like [1, 1, 1, 0, 1, 0] and at the output layer I have N neurons that outputs a ...
Kenenbek Arzymatov's user avatar
1 vote
1 answer
4k views

class weights formula for imbalanced dataset

I am trying to make some semantic segmentation. I have 7 imbalanced classes in my case. I found several methods for handling Class Imbalance in a dataset is to perform Undersampling for the Majority ...
safa's user avatar
  • 11
1 vote
1 answer
239 views

Update of mean and variance of weights

I'm trying to understand the Bayes by Backprop algorithm from the paper Weight Uncertainty in Neural Networks, the idea is to make a NN in which each weight has it's own probability distribution. I ...
Cosapocha's user avatar
1 vote
1 answer
508 views

Mathematical bias and weight vs machine learning bias and weight

I am a little confused about the term Bias and Weight with respect to machine learning. Say we want to predict the heights of people whose weights are given. So plot weights to x-axis and height to ...
Encipher's user avatar
  • 359
1 vote
0 answers
18 views

Can we talk about vanishing activations?

When updating the weights of a deep neural network using backpropagation, to update the weights of a given hidden layer, we use both the partial derivatives of the objective function with respect to ...
David Cian's user avatar
1 vote
1 answer
269 views

Matrix factorization how to initialize weights and biases?

I have a matrix factorization and I'm wondering how I should initialize its weights and biases. When getting prediction (recommendation), after computing a dot product and adding bias I want to use ...
Karol's user avatar
  • 111
3 votes
2 answers
78 views

Result of uniform weight initialization in all neurons

Background cs231n has the question regarding how to initialize weights. Question Please confirm or correct my understandings. I think the weight value will be the same in all the neurons with ReLU ...
mon's user avatar
  • 711
2 votes
1 answer
578 views

Why is it okay to set the bias vector up with zeros, and not the weight matrices?

We do not initialize weight matrices with zeros because the symmetry isn’t broken during the backward pass, and subsequently in the parameter updating process. But it is safe to set the bias vector up ...
truth's user avatar
  • 280
1 vote
1 answer
75 views

Initializing weights that are a pointwise product of multiple variables

In two-layer perceptrons that slide across words of text, such as word2vec and fastText, hidden layer heights may be a product of two random variables such as positional embeddings and word embeddings ...
Witiko's user avatar
  • 111
0 votes
1 answer
119 views

Does training of neural networks follow the same order in each epoch?

Each epoch uses the weight from the end of the previous epoch(correct me if I am wrong). Is the updating of parameters after each batch always in the same order? To rephrase, are the batches always in ...
Borut Flis's user avatar
0 votes
1 answer
29 views

Why don't different output weights break symmetry?

My deep learning lecturer told us that if a hidden node has identical input weights to another, then the weights will remain the same over the training/there will be no separation. This is confusing ...
Casebash's user avatar
  • 113
1 vote
1 answer
41 views

Weight-initialisation and model instability

Trying to calibrate a relatively vanilla NN, setting the hyper-parameter tuning aside*, it appears that weight initialisation has a lot of impact on the model output. Ie. Models calibrated with ...
Lucas Morin's user avatar
  • 2,196
1 vote
2 answers
242 views

What is wrong with a neural network model which is so dependent on the seed of initialization?

I have a fully-connected neural network with one hidden layer with 2 units which its goal is to classify a dataset with 324 samples and 300 features. 50% of the dataset is used for train and 50% of it ...
user137927's user avatar
1 vote
0 answers
103 views

Is the hyperbolic tangent function a solution to the weight clipping problem of WGAN?

Instead of clipping to the range $[-c,c]$ in WGAN (Wasserstein generative adversarial network), why not smoothly map into the range $[-c,c]$ by using $c\times \mathrm{tanh}(w)$? This would guarantee ...
crow's user avatar
  • 1
1 vote
1 answer
262 views

Should weight distribution change more when fine-tuning transformers-based classifier?

I'm using pre-trained DistilBERT model from Huggingface with custom classification head, which is almost the same as in the reference implementation: ...
Marcin Zablocki's user avatar
4 votes
2 answers
995 views

Compare Coefficients of Different Regression Models

in my project, I am using asuite of shallow and deep learning models in order to see which has the best performance on my data. However, in the pool of shallow machine learning models, I want to be ...
Perl Del Rey's user avatar
7 votes
1 answer
2k views

Is it wrong to use Glorot Initialization with ReLu Activation?

I'm reading that keras' default initialization is glorot_uniform. However, all of the tutorials I see are using relu ...
Kermit's user avatar
  • 529
1 vote
0 answers
42 views

How could we use a neuron instead of 100 neurons in 0.01 standard Weight initialization

Many books explain that we don't need to use 100 neurons and just can use a neuron if all result values(sigmoid(Wx)) are same at each of hidden layers. but, I don't know that we can just use a ...
douner's user avatar
  • 115
1 vote
0 answers
88 views

Cold-start problem in Real Time Bidding

I'm currently on the reading stage of the deployment of an RTB system. I've seen the problem of a cold start (having no initial guess of how to bid) in several papers, but I haven't really seen it be ...
Bananin's user avatar
  • 111
0 votes
2 answers
39 views

Why is the variance going down so much in this weight initialization problem(using pytorch)?

first look at this example >>> x = t.randn(512) >>> w = t.randn(512, 500000) >>> (x @ w).var() tensor(513.9548) it makes sense that ...
katiex7's user avatar
  • 199
1 vote
1 answer
30 views

Same probability for all classes

I implemented a fully connected MLP of shape [783 (input), 128 (hidden layer) and 10 (output)] the hidden layer had a sigmoid activation function and the output a sofmax. I tested with the dataset of ...
Agustin Barrachina's user avatar
1 vote
1 answer
252 views

How can I have the same initialization for all my networks?

I want to have the same weights for layer initializations in all my networks, so that when I'm comparing their first epoch loss they all start from the same value. Is there a way in keras to do this? ...
Moeinh77's user avatar
  • 221
2 votes
1 answer
507 views

A model that only works by setting all initial weights to zero

In this model from MusicNet, they set the initial weights of their neural network to all zeros. ...
Raven Cheuk's user avatar
2 votes
3 answers
678 views

weight training speed too slow in CNNs

I'm writing my own CNN code from scratch. Though I got fast, converged and satisfactory results, the trained weights change very little in value (while cost/loss function drops in time rapidly in a ...
feynman's user avatar
  • 237
3 votes
1 answer
431 views

How large of a value should a weight have in a neural network?

If you're assigning random values to the weights in a neural network before back-propagation, is there a certain maximum or minimum value for each weight ( for example, 0 < w < 1000 ) or can ...
PickleDonk's user avatar
1 vote
1 answer
151 views

Weights initialization in Neural Network

I was viewing code for custom neural network for sentiment analysis. It had 3 layers (1 hidden layer). I am more concerned with weight initialization for the layers ...
Sagar Dhungel's user avatar
0 votes
2 answers
220 views

C++ return array from function

I would like to implement machine learning algorithm in C++ without using any C++ machine learning library. So I'm writing this initializer function for generating zero matrices but can't figure out ...
Rouh Al Noor Auritro's user avatar
6 votes
2 answers
340 views

What are the reasons for drawing initial neural network weights from the Gaussian distribution?

Are there theoretical or empirical reasons for drawing initial weights of a multilayer perceptron from a Gaussian rather than from, say, a Cauchy distribution?
Stephane Bersier's user avatar
10 votes
2 answers
18k views

what is difference between the DDQN and DQN?

I think I did not understand what is the difference between DQN and DDQN in implementation. I understand that we change the traget network during the running of DDQN but I do not understand how it is ...
user10296606's user avatar
  • 1,834
7 votes
2 answers
9k views

What are the cases where it is fine to initialize all weights to zero

I've taken a few online courses in machine learning, and in general, the advice has been to choose random weights for a neural network to ensure that your neurons don't all learn the same thing, ...
Stephen's user avatar
  • 272