Questions tagged [optimization]

In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

Filter by
Sorted by
Tagged with
0
votes
2answers
39 views

Confused between optimizer and loss function

I always thought the SGD was a loss function then I read this on a notebook ...
0
votes
1answer
21 views

Why use gradient descent on Deep Nets / RNNs when cost function is not convex?

Why do we use gradient descent on very non-convex loss functions such as in Deep nets / RNNs rather than a heuristic search (genetic algorithms, simulated annealing, etc)?
3
votes
1answer
123 views

Which learning rate should I choose?

I'm training a segmentation model, Unet++, on 2d images and I am now trying to find the optimal learning rate. The backbone of the model is Resnet34, I use Adam optimizer and the loss function is the ...
1
vote
1answer
29 views

a neural network can be used as an optimizer?

I was wondering if a neural network can be used as an optimizer. In other words, a network2 used as an optimizer takes the loss value of network1 and based on that it predicts the best weights for ...
4
votes
1answer
38 views

Need to kickstart learning rates

I was just looking at the docs on Pytorch for different available schedulers and I found one that I am having some trouble understanding here. The others seem to make sense: As training progresses, ...
0
votes
0answers
20 views

Does it make sense to use a learning rate scheduler with an adaptive learning rate method?

I'm training a large variety of networks and found out that, while using Adadelta as an optimizer, applying also the keras callback ...
1
vote
0answers
25 views

How to balance time/effort with transformations, feature selection, and models efficacy in nlp? [closed]

Edit: Question has been edited for reopening (see comment section for justification) Being to new text analytics, I haven't gotten the hang of navigating a typical workflow given the longer times ...
0
votes
0answers
23 views

optimizing customization: beyond a combinatorial approach?

Say I have a generic non-linear model (ANN, Random Forest, Gradient Boosting, etc) that wants, based on a set of features (price of a product, service duration, age, etc), to give me a prediction of ...
1
vote
0answers
24 views

Why is the exploding/vanishing gradient problem not solved by line search?

The problem of vanishing gradients is basically that since our step size is proportional to the gradient, if the gradient is very small, it might take a long time to reach a local minimum. So why don'...
0
votes
1answer
14 views

use the same gradient to maximize one part of the model and minimize another part of the same model

I want to calculate the gradient and use the same gradient to minimize one part and maximize another part of the same network (kind of adversarial case). For me, Ideal case would be, if there are two ...
0
votes
1answer
155 views

Why sparse features should have bigger learning rates associated? And how Adagrad achieves this? [closed]

I was learning about Adagrad optimizer. I came to know that it has a very helpful functionality which is that we can have lower learning rates for the features that are more common and greater ...
0
votes
1answer
52 views

Wouldn't it make more sense to give less importance to gradient far away in past in AdaGrad? [closed]

This is the update equation of a weight by AdaGrad: $$w_{new} = w_{old} - \frac{lr}{\sqrt{G_{}+E}}.G_{w_{old}}$$ Where $G$ is the sum of the gradients of the same weight at previous iterations, $E$ is ...
0
votes
1answer
55 views

Why are we taking the square root of the gradient in Adagrad? [closed]

This is how we update weights with Adagrad: $$w_i = w_i - \frac{lr}{\sqrt{g_i+E}}$$ where, $w_i$ is the $i^{th}$ weight, $lr$ is the learning rate, $g_i$ is the gradient of the $i^{th}$ weight at all ...
2
votes
4answers
159 views

Is it possible to get worse model after optimization?

I am trying recently to optimize models but for some reason, whenever I try to run the optimization the model score in the end is worse than before, so I believe I do something wrong. in order to ...
0
votes
1answer
20 views

Back Propagation Vs Learning rate in Neuralnet Optimisation

I was doing some research on how backpropagation works? I read that, backpropagation is used to find the optimal weight of each neuron after every iteration using partial derivates and updates the ...
1
vote
1answer
29 views

Good chromosome representation in a VRPTW genetic algorithm

I have a genetic algorithm for a vehicle routing problem with time windows and I need to implement certain modifications. I am not sure what would be the best chromosome representations. I have tasks ...
2
votes
1answer
182 views

When is Non-Stochastic Global Optimization Preferable or Necessary?

Background I'm specifically referring to non-convex black-box optimization problems of the form: $ \text{min} f(\vec{x})$ $s.t. \ \ a_i\le x_i \le b_i \ \forall i\in \{1,2,...,n\} \ \ \ \text{and}\ \ ...
0
votes
1answer
45 views

How To Motivate A Neural Network

Suppose a training dataset contains the following inputs: company size number of employees turnover average salary country years of operation ...and outputs: ...
1
vote
0answers
134 views

Why does Faster R-CNN use SGD optimizer instead of Adam?

I just start learning Faster R-CNN and I have some doubts about the optimizer of this network. In my understanding, Adam optimizer performs much better than SGD in a lot of networks. However, the ...
0
votes
0answers
16 views

Implementation of RMS prop for linear regression

I'm trying to implement linear regression using Rms Prop optimizer from scratch. Code: ...
3
votes
1answer
49 views

When does it make sense to choose gradient descent for SVM over liblinear?

I understand using gradient descent methods with SVM is intractable if you've used the kernel trick. In that case, best to use libsvm as your solver. But in the case that you are not using a kernel ...
4
votes
2answers
96 views

Does convergence of loss function is always guarnteed?

Which of the following is true, given the optimal learning rate? (i) For convex loss functions (i.e. with a bowl shape), batch gradient descent is guaranteed to eventually converge to the global ...
-1
votes
3answers
56 views

Why training of a neural network will require multiple iterations? [closed]

I can't understand why training of a neural network will require multiple iterations (theoretically)? Can anyone explain why, please?
0
votes
0answers
61 views

How does Batch normalization help optimization? Proof

I am reading the paper How Does Batch Normalization Help Optimization found here. $\newcommand{\norm}[1]{\left\lVert#1\right\rVert}$ But I am having trouble understanding the proof of the paper. It's ...
0
votes
1answer
29 views

“Invalid value” in RMSprop implementation from scratch in Python

Edit 2: The regularization term (reg_term) is sometimes negative due negatative parameters. Hence S[f"dW{l}"] contains some negative values. I realize the reg_term has to be added before ...
1
vote
1answer
26 views

Support Vector Machines with soft margin: solving the dual form

I am currently struggling with finding an analytical solution for the $\alpha_k$. I have derived the following constrained optimization problem: $$ L = \sum_{i=1}^{N} \alpha_i - \frac{1}{2} \sum_{i=1}^...
1
vote
0answers
12 views

Using Iterative Hard/Soft Thresholding in autoencoder with non linear activation

Can someone please give an intuitive explanation of the difference between the Iterative Hard Thresholding VS Iterative Soft thresholding algorithm? And if we can use these algorithms in an ...
2
votes
1answer
29 views

how to find the best parameters to solve a differential equation?

I have a differential equation: def func(Y, t, r, p, K, alpha): return r * (Y ** p) * (1 - (Y / K) ** alpha) and I want to find the best parameters that fit (r,...
2
votes
1answer
58 views

Setting BATCH SIZE when performing multi-class classification with imbalanced dataset

I have a question regarding BATCH_SIZE on multi-class classification task with imbalanced data. I have 5 classes and a small dataset of around ...
2
votes
2answers
54 views

Is it reasonable to train a neural network many times and cherry pick the best result based on test dataset accuracy?

My current advisor at Uni insists that I train 10 instances of the same network and pick the one with best test accuracy in order to escape the "local minima". In my opinion this does not ...
1
vote
1answer
23 views

What is the best way to pick the optimized configuration from this dataset?

I have about 8000 configurations in an excel sheet. each configuration has four scores as seen in the image below. I would like to choose the best solution that has the highest lighting level score, ...
0
votes
0answers
23 views

Analytical solution for optimization with inequality constraints

Let the following be known matrices with dimensions as: $M = nXk$, all elements >0 $w_b = 1Xn$, all elements >0 , sums to 1 $S = nXn$, a positive semidefinite matrix $C_c = 1Xk$, all elements &...
3
votes
0answers
48 views

Scipy minimization failing with inequality constraints or bounds [closed]

I am trying to use scipy.optimize to solve a minimization problem but getting failures on using an inequality constraint or a bound. Looking for any suggestions regarding proper usage of constraints ...
2
votes
2answers
29 views

how classification scores are interpreted?

I would like to know how to interpret classification scores (i am not sure about the word score or probability, please correct me). For example, for a binary classification positive values are ...
1
vote
1answer
30 views

How does Pytorch deal with non-differentiable activation functions during backprop?

I've read many posts on how Pytorch deal with non-differentiability in the network due to non-differentiable (or almost everywhere differentiable - doesn't make it that much better) activation ...
2
votes
1answer
105 views

How similar is Adam optimization and Gradient clipping?

According to the Adam optimization update rule: $$m \leftarrow \beta_1 m + (1 - \beta_1)\nabla J(\theta)$$ $$v \leftarrow \beta_2 v + (1 - \beta_2)(\nabla J(\theta) \odot \nabla J(\theta))$$ $$\theta \...
0
votes
1answer
54 views

Oracle in optimization

I have encountered the word oracle in the following context: Given an $\alpha$-approximate oracle for stochastic optimization we show how to implement an $\alpha$-approximate solution for robust ...
3
votes
2answers
233 views

Difference between RMSProp and Momentum?

Can someone please tell me the clear difference between the approaches of RMSProp and Gradient Descent with Momentum ? Both try to achieve the same effect . One of the blogs that I read states the ...
1
vote
1answer
30 views

Reversing the input and output of an ML algorithm to Optimize

My dataset consists of multiple input variables (X) and multiple output variables (Y). For example: ...
3
votes
1answer
198 views

SGD versus Adam Optimization Clarification

Reading the Adam paper, I need some clarificaiton. It states that SGD optimization updates the parameters with the same learning rate (i.e. it does not change throughout training). They state Adam is ...
0
votes
0answers
19 views

When scaled conjugate gradient backpropagation is suitable for backpropagation?

I usually use gradient descent with Adam optimizer to perform backpropagation in deep learning methods. I knew it is a very efficient method. The question is in which situations we can use "scaled ...
0
votes
0answers
35 views

Save and restore custom optimizers for continued training of neural networks in TensorFlow

My question is essentially the exact same as that specified here but without using the Keras backend. Namely, how does one save and restore custom optimizers to their last state in TensorFlow (e.g. <...
0
votes
0answers
17 views

Logistic loss increasing while training with minibatches using the adam algorithm

I am trying to write my own code to use the adam algorithm for logistic regression. I am pretty sure It is training correctly as when I run it I am able to accurately classify a bunch of toy data that ...
1
vote
2answers
65 views

Appropriate loss function for multi-hot output vectors

I have some data in which model inputs and outputs (which are the same size) belong to multiple classes concurrently. A single input or output is a vector of zeros somewhere between one and four ...
1
vote
1answer
85 views

Google OR-Tools - Routing - Penalties - Adding Different Penalty to Different Location (Python)

I am using Google's OR-Tools for route optimisation. References can be found here. I am performing an optimisation where certain pick-up locations are dropped based on a penalty at each location. The ...
2
votes
2answers
63 views

Formal math notation of masked vector

I'm struggling to write my algorithm in a concise and correct way. The following is an explanation for an optimizer's update step of part of a vector of weights (not a matrix in my case). I have a ...
0
votes
1answer
31 views

Cat Classifier becomes worse the more you train it

I am using a dataset from kaggle to train a feed forward neural-neteork with no convolutional layers. I wanted to try it this was as a learning exercise with Pytorch without Transfer Learning and ...
0
votes
0answers
9 views

Simplifying Log Likelihood equation

I am reading through a paper (https://www.mitpressjournals.org/doi/pdf/10.1162/0891201053630273) where they describe logloss as a ranking function and can be simplified to the margin of the training ...
0
votes
0answers
11 views

Best ways of encoding neural networks for GA

I am trying to create a genetic algorithm that should create optimal neural networks based on two parameters - network size and value of the fitness function, so that we can find networks that are ...
0
votes
0answers
31 views

How to compute error (instead of gradient of error) for each node in backpropagation?

I was going through the relevant chain rule mathematics and I have successfully implemented backpropagation from scratch for MNIST (once, I even tried doing this for a small sample data I created by ...

1
2 3 4 5
7