The tag has no usage guidance.

36 questions
Filter by
Sorted by
Tagged with
172 views

Currently I am working on a custom fine-tune of several code LLMs and while working on the DeepSeekCoder I encountered a strange behaviour. When training the model earlier or later the loss goes to ...
• 1
7 views

### How to derive the formula 13 in the Xavier Initialization paper

How to derive the formula 13 in the Xavier Initialization paper Understanding the difficulty of training deep feedforward neural networks from the formula 6?
• 771
1 vote
63 views

### Use of Gradient with respect to feature instead of model parameters

Generally, for any machine learning/deep learning system, we compute a loss, $L = l(x, \theta, y)$ which is a function of the input feature vector $x$ (after activation), model parameters $\theta$ (...
362 views

### Gradient Boosting - Why pseudo-residuals?

I have some questions I don't really understand regarding the Gradient Boosting algorithm with Decision Trees: Does the initial value matter as $\hat{y}$ or could you pick any, f.e between 0 and 1? ...
• 11
31 views

### Does learning rate depend on input and output range?

I watched hours of videos on gradient descent and still feel pretty confused. Let's say I have a "model": y = x * w I use 2 as my target ...
• 103
22 views

### why do we get negative predictions for boosting model even if the target variables are strictly positive value?

why do we get negative predictions for boosting model even if the target variables are strictly positive value? I read another thread but I don't understand the explanation.
2k views

### How to correctly create a PyTorch Tensor from a Pandas DataFrame?

I have loaded my data into a Pandas DataFrame, and performed some pre-processing, and then I need to convert it into a PyTorch Tensor for training as my features data. Obviously, This new tensor do ...
• 101
1 vote
41 views

### How do we derive our loss function from the gradient objective?

I've been dwelling through RL theory and practice and one particular part I find hard to properly understand is the relation between the practical loss function and ...
• 111
146 views

### calculating derivative of bias in backpropagation

Looking at the algorithm in wikipedia, we can implement backpropagation by calculating: $$\delta^{L}=\left(f^{L}\right)'\cdot\nabla_{a^{L}}C$$ (where I treat $\left(f^{L}\right)'$ as an $n\times n$ ...
• 101
568 views

### Why would we add regularization loss to the gradient itself in an SVM?

I'm doing CS 231n on my own. I'm looking at this solution to a question that implements a SVM. Relevant code: ...
• 125
1 vote
16 views

### Gradient Ascent and directional derivative

Suppose that you want to estimate a local maximum of the real function $f(x,y,z)$ with gradient ascent. Given a starting point $(x_0, y_0, z_0)$, the approach is to compute the gradient at this ...
• 105
1 vote
48 views

2k views

### Gradient of a function in Python

I've defined a function in this way: def qfun(par): return(par[0]+atan(par[3])*par[1]+atan(par[4])*par[2]) How can I obtain the gradient of this function ...
• 150
64 views

### How can we get gradient with some other loss function apart from MSE?

In most of the gradient search examples, the update to weights are done by subtracting the derivative of MSE. Can we have an example, where we did not use ...
1 vote
37 views

### Matlab Optimization. Meaning of warning: "The slope should be 2. It appears to be 1."

I'm using the manopt package to solve some optimization problems in matlab. The problem is of the form. problem.cost = @(x) f(x) problem.egrad = @(x) g(x) After the problem definition, I check ...
• 111
97 views

### Gradient Checking: MeanSquareError. Why huge epsilon improves discrepancy?

I am using custom C++ code, and coded a simple "Mean Squared Error" layer. Temporarily using it for the 'classification task', not a simple regression. ...maybe this causes the issues? I don't have ...
• 2,756
22k views

What's causing the vanishing gradient or exploding gradient, and what are the measures to be taken to prevent it?
• 159
369 views

ReLU is used as an activation function that serves two purposes: Breaking linearity in DNN. Helping in handling Vanishing Gradient problem. For Exploding Gradient problem, we use Gradient Clipping ...
• 1,272