# Tag Info

1

One small correction- G here is the sum of squares of the gradient of a particular parameter which is kept track of and is monotonic in nature. That being said, the way I see the equation functions is, it lets the past dictate the future or simply put, the past gradients of the parameter decides by how much it's corresponding learning rate drops. In SGD no ...

-1

We are taking the square root of $G_i$ because $G_i$ is the sum of squares of the gradient of weight $i$ from iterations $1$ to the current iteration. Since we squared all the gradients of all timesteps, $G_i$ was very large, and so we take its square root. Now you may think that, then why don't we just add the gradients instead of squaring the gradients, ...

0

You can find a short explanation here and a detailed explanation here.

0

You are using (stochastic) gradient descent. For that to work properly, the learning rate (step size) must be set correctly. I assume that the error lies there. Instead, you could try logistic regression via IRLS (see its definition), compare also IRLS vs GD Or for the input you tested, you just found a bad local optimum.

0

you can have a look at the gradient visualization method proposed in this article: We present a simple visualization method based on “filter normalization.” The sharpness of minimizers correlates well with generalization error when this normalization is used, even when making comparisons across disparate network architectures and training methods. This ...

2

It is slower in terms of time necessary to compute one full epoch. BUT it is faster in terms of convergence i.e. how many epochs are necessary to finish training which is what you care about at the end of the day. It is because you take many gradient steps to the optimum in one epoch when using batch/stochastic GD while in GD you only take one step per epoch....

1

1 - The computation time of SGD is much lower than GD as you only use a subset of the whole data, that is why it is actually faster (time-wise) even though it seems you do more stuff. 2- With GD you compute your gradient on all the data you have, therefore the computed gradient gives you the best direction to minimize your function on the whole dataset. With ...

3

That's because the whole loss is $\frac{1}{N} \sum\limits_{i=1}^N L(x_i, y_i)$ and that number $N$ is the dataset size, it can be very large. It's just too slow to compute the true gradient, thus we compute its unbiased estimate via Monte Carlo. There are some theorems that say that stochastic gradient descent converges under certain conditions, so it's a ...

0

Probably in the 1-dimensional case this more precise class of functions you aim at would be something like: Piece-wise monotonous (thus approachable with gradient descent locally) with up to one flip (thus you know that one optimum would be where the pieces meet, the other would be at the extremes) However, even if one is able to come up with such a ...

1

The behavior appears to actually depend on the learning rate $\eta$; a smaller $\eta$ affects which points are misclassified in the next iteration, which affects the weight update more than just by the simple scaling you alluded to. With appropriately small learning rates though, it seems you are guaranteed convergence to some local minimum, if you avoid ...

1

As mentioned in the comment, if the question is when does it make sense to use coordinate descent over stochastic gradient descent, then, one advantage with coordinate descent is that, it updates only one parameter at a time. Thus, when the data has a very very large number of features, it might make sense to use Coordinate Descent over stochastic gradient ...

0

Since you are asking which packages, H2o GBM, Rpart and R gbm handle missing as well. Through different ways such as surrogate variables, another category, 3-way split (left, right, missing). You should also ask what if there is missing in a feature during scoring that had no missing during training - how will that be handled? Just handling missing during ...

1

LightGBM by default handles null values by setting them to zero. You can also have it assume zeros are null values by setting zero_as_missing=true. So, while it handles them in the back end, it doesn't do anything different to you imputing with zeros. here is the documentation : https://lightgbm.readthedocs.io/en/latest/Advanced-Topics.html Personally, I ...

1

As you and @gunes pointed out in this post, your formula are correct, but hyperparameters $\alpha$ and $iterations$ were not well adjusted.

Top 50 recent answers are included