Questions tagged [weight-initialization]
The weight-initialization tag has no usage guidance.
23
questions
0
votes
0answers
15 views
Is the hyperbolic tangent function a solution to the weight clipping problem of WGAN?
Instead of clipping to the range [-c,c] why not smoothly map into the range [-c,c] by using c*tanh(w) ? This would guarantee the Lipschizt constant is no greater than c.
The problem I am talking ...
1
vote
1answer
18 views
Should weight distribution change more when fine-tuning transformers-based classifier?
I'm using pre-trained DistilBERT model from Huggingface with custom classification head, which is almost the same as in the reference implementation:
...
0
votes
0answers
11 views
How to make sure that the learned weights are initialized instead of only layer structure?
I have trained a model for 100 epochs. The network is designed to save checkpoints after every 10th epoch. Besides this, once the training finished I saved the model using these commands:
...
4
votes
2answers
202 views
Compare Coefficients of Different Regression Models
in my project, I am using asuite of shallow and deep learning models in order to see which has the best performance on my data. However, in the pool of shallow machine learning models, I want to be ...
3
votes
1answer
46 views
Is it wrong to use Glorot Initialization with ReLu Activation?
I'm reading that keras' default initialization is glorot_uniform.
However, all of the tutorials I see are using relu ...
1
vote
0answers
40 views
How could we use a neuron instead of 100 neurons in 0.01 standard Weight initialization
Many books explain that we don't need to use 100 neurons and just can use a neuron if all result values(sigmoid(Wx)) are same at each of hidden layers.
but, I don't know that we can just use a ...
1
vote
0answers
21 views
Cold-start problem in Real Time Bidding
I'm currently on the reading stage of the deployment of an RTB system. I've seen the problem of a cold start (having no initial guess of how to bid) in several papers, but I haven't really seen it be ...
0
votes
0answers
21 views
What does Keras Initializers assume about the datasets distribution?
I have a dataset that consists of 9000 Synthetic Aperture Radar images normalized to [0,1]. One of them is visualized in the histogram below. All of the images follow the same structure where most of ...
0
votes
1answer
15 views
Why is the variance going down so much in this weight initialization problem(using pytorch)?
first look at this example
>>> x = t.randn(512)
>>> w = t.randn(512, 500000)
>>> (x @ w).var()
tensor(513.9548)
it makes sense that ...
1
vote
1answer
16 views
Same probability for all classes
I implemented a fully connected MLP of shape [783 (input), 128 (hidden layer) and 10 (output)] the hidden layer had a sigmoid activation function and the output a sofmax.
I tested with the dataset of ...
0
votes
0answers
32 views
problem with vanishing/exploding gradient problems solution
I have few doubts around vanishing/exploding gradients.
The problem with vanishing gradient is, When the weights are randomly initialized in a deep network, During back propagation initial layers ...
1
vote
1answer
33 views
How can I have the same initialization for all my networks?
I want to have the same weights for layer initializations in all my networks, so that when I'm comparing their first epoch loss they all start from the same value. Is there a way in keras to do this?
...
0
votes
0answers
22 views
How to select initializers for DDQN in Keras?
I'm using DDQN for OpenAI Gym games (like CartPole, MountainCar). It occurred to me that the weight/bias initialization might have a reasonable impact on how quickly the network starts to learn so I ...
0
votes
0answers
149 views
save initial weights of model in keras
I am trying to save initial weights of the autoencoder model. I was verifying if I have saved them properly by two methods. In the first method I have used autoencoder.get_weights() and in the ...
0
votes
0answers
12 views
Initialisation of weights for input sensitivity
Iam trying to architect a neural network. I have 3 inputs and I require the output to be as sensitive as possible to changes in the input. The outputs are calculated using 10 hidden ReLU layers and a ...
1
vote
0answers
33 views
A model that only works by setting all initial weights to zero
In this model from MusicNet, they set the initial weights of their neural network to all zeros.
...
2
votes
3answers
155 views
weight training speed too slow in CNNs
I'm writing my own CNN code from scratch. Though I got fast, converged and satisfactory results, the trained weights change very little in value (while cost/loss function drops in time rapidly in a ...
3
votes
1answer
54 views
How large of a value should a weight have in a neural network?
If you're assigning random values to the weights in a neural network before back-propagation, is there a certain maximum or minimum value for each weight ( for example, 0 < w < 1000 ) or can ...
1
vote
1answer
72 views
Weights initialization in Neural Network
I was viewing code for custom neural network for sentiment analysis. It had 3 layers (1 hidden layer). I am more concerned with weight initialization for the layers
...
0
votes
2answers
141 views
C++ return array from function
I would like to implement machine learning algorithm in C++ without using any C++ machine learning library. So I'm writing this initializer function for generating zero matrices but can't figure out ...
2
votes
0answers
29 views
What are the reasons for drawing initial neural network weights from a Gaussian?
Are there theoretical or empirical reasons for drawing initial weights of a multilayer perceptron from a Gaussian rather than from, say, a Cauchy distribution?
4
votes
2answers
4k views
what is difference between the DDQN and DQN?
I think I did not understand what is the difference between DQN and DDQN in implementation.
I understand that we change the traget network during the running of DDQN but I do not understand how it is ...
4
votes
2answers
3k views
What are the cases where it is fine to initialize all weights to zero
I've taken a few online courses in machine learning, and in general, the advice has been to choose random weights for a neural network to ensure that your neurons don't all learn the same thing, ...
