Questions tagged [optimization]

In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

Filter by
Sorted by
Tagged with
0
votes
0answers
8 views

use the same gradient to maximize one part of the model and minimize another part of the same model

I want to calculate the gradient and use the same gradient to minimize one part and maximize another part of the same network (kind of adversarial case). For me, Ideal case would be, if there are two ...
-3
votes
0answers
27 views

Will the total number of $s_t$ variables be dependent on the length of dataset in AdaGrad? [closed]

We update a weight $w$ with Adagrad like this: $$s_t=s_{t-1}+(g_t)^2$$ $$w_t=w_{t-1}-\frac{lr}{\sqrt{s_t+e}}*g_t$$ Where, $lr$ is the learning rate, $e$ is to prevent division by zero, $g_t$ is the ...
-2
votes
1answer
40 views

Wouldn't AdaGrad fail in this case? [closed]

This is the update rule for AdaGrad: $$s_t=s_{t−1}+(g_t)^2$$ $$w_t=w_{t−1}−\frac{lr}{\sqrt{s_t+e}}∗g_t$$ Now, let's say we have a data set which has 90 non-sparse features (i.e. features which has ...
-1
votes
0answers
35 views

Is this the reason why we don't just divide the learning rate by the gradient in AdaGrad? [closed]

In AdaGrad, the update equation is this: $$s_t = s_{t-1}*(G_t)^2$$ $$w_t = w_{t-1} *\frac{lr}{\sqrt{s_t+e}}*G_t$$ Where, $G_t$ is the gradient at time $t$ and $lr$ is the learing rate, and $e$ is to ...
-1
votes
0answers
26 views

Why do we use AdaGrad optimizer with sparse features? [closed]

I have heard people saying that AdaGrad optimizer is best-suited with sparse features. But, why is that?
-2
votes
0answers
23 views

How does the AdaGrad optimizer obtain it's following fuctionality? [closed]

I was learning about AdaGrad optimizer. I came to know that it has a functionality which is that it gives lower learning rates to the features that are more common and higher learning rates to the ...
-1
votes
1answer
84 views

Why sparse features should have bigger learning rates associated? And how Adagrad achieves this? [closed]

I was learning about Adagrad optimizer. I came to know that it has a very helpful functionality which is that we can have lower learning rates for the features that are more common and greater ...
0
votes
1answer
50 views

Wouldn't it make more sense to give less importance to gradient far away in past in AdaGrad? [closed]

This is the update equation of a weight by AdaGrad: $$w_{new} = w_{old} - \frac{lr}{\sqrt{G_{}+E}}.G_{w_{old}}$$ Where $G$ is the sum of the gradients of the same weight at previous iterations, $E$ is ...
0
votes
1answer
42 views

Why are we taking the square root of the gradient in Adagrad? [closed]

This is how we update weights with Adagrad: $$w_i = w_i - \frac{lr}{\sqrt{g_i+E}}$$ where, $w_i$ is the $i^{th}$ weight, $lr$ is the learning rate, $g_i$ is the gradient of the $i^{th}$ weight at all ...
0
votes
4answers
124 views
+50

Is it possible to get worse model after optimization?

I am trying recently to optimize models but for some reason, whenever I try to run the optimization the model score in the end is worse than before, so I believe I do something wrong. in order to ...
0
votes
1answer
16 views

Back Propagation Vs Learning rate in Neuralnet Optimisation

I was doing some research on how backpropagation works? I read that, backpropagation is used to find the optimal weight of each neuron after every iteration using partial derivates and updates the ...
-1
votes
0answers
20 views

Effect of label normalization on optimization?

Let's say in a regression task I have a range of labels 1-60. If I normalize the labels and squeeze those into 0-1 range (by dividing 60) and calculate loss then the calculated loss will be very small ...
0
votes
0answers
11 views

How to use keras to find best solution using smallest time as score

So i have a dataset of simulations that have a bunch of parameters that are of 2 groups, lets call them config_params and results_params. For every simulation I also have a time of running. I would ...
1
vote
1answer
25 views

Good chromosome representation in a VRPTW genetic algorithm

I have a genetic algorithm for a vehicle routing problem with time windows and I need to implement certain modifications. I am not sure what would be the best chromosome representations. I have tasks ...
3
votes
1answer
143 views

When is Non-Stochastic Global Optimization Preferable or Necessary?

Background I'm specifically referring to non-convex black-box optimization problems of the form: $ \text{min} f(\vec{x})$ $s.t. \ \ a_i\le x_i \le b_i \ \forall i\in \{1,2,...,n\} \ \ \ \text{and}\ \ ...
0
votes
1answer
43 views

How To Motivate A Neural Network

Suppose a training dataset contains the following inputs: company size number of employees turnover average salary country years of operation ...and outputs: ...
0
votes
1answer
40 views

Optimization Terminology: Non-Stochastic versus Deterministic

Background from Wikipedia: deterministic global optimizers "provide theoretical guarantees that the reported solution is indeed the global one, within some predefined tolerance. The term ...
0
votes
0answers
38 views

Why does Faster R-CNN use SGD optimizer instead of Adam?

I just start learning Faster R-CNN and I have some doubts about the optimizer of this network. In my understanding, Adam optimizer performs much better than SGD in a lot of networks. However, the ...
0
votes
0answers
15 views

Implementation of RMS prop for linear regression

I'm trying to implement linear regression using Rms Prop optimizer from scratch. Code: ...
3
votes
1answer
43 views

When does it make sense to choose gradient descent for SVM over liblinear?

I understand using gradient descent methods with SVM is intractable if you've used the kernel trick. In that case, best to use libsvm as your solver. But in the case that you are not using a kernel ...
4
votes
2answers
76 views

Does convergence of loss function is always guarnteed?

Which of the following is true, given the optimal learning rate? (i) For convex loss functions (i.e. with a bowl shape), batch gradient descent is guaranteed to eventually converge to the global ...
-1
votes
3answers
44 views

Why training of a neural network will require multiple iterations? [closed]

I can't understand why training of a neural network will require multiple iterations (theoretically)? Can anyone explain why, please?
0
votes
0answers
55 views

How does Batch normalization help optimization? Proof

I am reading the paper How Does Batch Normalization Help Optimization found here. $\newcommand{\norm}[1]{\left\lVert#1\right\rVert}$ But I am having trouble understanding the proof of the paper. It's ...
0
votes
1answer
24 views

“Invalid value” in RMSprop implementation from scratch in Python

Edit 2: The regularization term (reg_term) is sometimes negative due negatative parameters. Hence S[f"dW{l}"] contains some negative values. I realize the reg_term has to be added before ...
1
vote
1answer
22 views

Support Vector Machines with soft margin: solving the dual form

I am currently struggling with finding an analytical solution for the $\alpha_k$. I have derived the following constrained optimization problem: $$ L = \sum_{i=1}^{N} \alpha_i - \frac{1}{2} \sum_{i=1}^...
1
vote
0answers
12 views

Using Iterative Hard/Soft Thresholding in autoencoder with non linear activation

Can someone please give an intuitive explanation of the difference between the Iterative Hard Thresholding VS Iterative Soft thresholding algorithm? And if we can use these algorithms in an ...
2
votes
1answer
29 views

how to find the best parameters to solve a differential equation?

I have a differential equation: def func(Y, t, r, p, K, alpha): return r * (Y ** p) * (1 - (Y / K) ** alpha) and I want to find the best parameters that fit (r,...
2
votes
0answers
36 views

Setting BATCH SIZE when performing multi-class classification with imbalanced dataset

I have a question regarding BATCH_SIZE on multi-class classification task with imbalanced data. I have 5 classes and a small dataset of around ...
2
votes
2answers
44 views

Is it reasonable to train a neural network many times and cherry pick the best result based on test dataset accuracy?

My current advisor at Uni insists that I train 10 instances of the same network and pick the one with best test accuracy in order to escape the "local minima". In my opinion this does not ...
1
vote
1answer
22 views

What is the best way to pick the optimized configuration from this dataset?

I have about 8000 configurations in an excel sheet. each configuration has four scores as seen in the image below. I would like to choose the best solution that has the highest lighting level score, ...
0
votes
0answers
20 views

Analytical solution for optimization with inequality constraints

Let the following be known matrices with dimensions as: $M = nXk$, all elements >0 $w_b = 1Xn$, all elements >0 , sums to 1 $S = nXn$, a positive semidefinite matrix $C_c = 1Xk$, all elements &...
3
votes
0answers
33 views

Scipy minimization failing with inequality constraints or bounds [closed]

I am trying to use scipy.optimize to solve a minimization problem but getting failures on using an inequality constraint or a bound. Looking for any suggestions regarding proper usage of constraints ...
2
votes
2answers
28 views

how classification scores are interpreted?

I would like to know how to interpret classification scores (i am not sure about the word score or probability, please correct me). For example, for a binary classification positive values are ...
0
votes
1answer
25 views

How does Pytorch deal with non-differentiable activation functions during backprop?

I've read many posts on how Pytorch deal with non-differentiability in the network due to non-differentiable (or almost everywhere differentiable - doesn't make it that much better) activation ...
2
votes
1answer
68 views

How similar is Adam optimization and Gradient clipping?

According to the Adam optimization update rule: $$m \leftarrow \beta_1 m + (1 - \beta_1)\nabla J(\theta)$$ $$v \leftarrow \beta_2 v + (1 - \beta_2)(\nabla J(\theta) \odot \nabla J(\theta))$$ $$\theta \...
0
votes
1answer
52 views

Oracle in optimization

I have encountered the word oracle in the following context: Given an $\alpha$-approximate oracle for stochastic optimization we show how to implement an $\alpha$-approximate solution for robust ...
3
votes
2answers
144 views

Difference between RMSProp and Momentum?

Can someone please tell me the clear difference between the approaches of RMSProp and Gradient Descent with Momentum ? Both try to achieve the same effect . One of the blogs that I read states the ...
1
vote
1answer
27 views

Reversing the input and output of an ML algorithm to Optimize

My dataset consists of multiple input variables (X) and multiple output variables (Y). For example: ...
3
votes
1answer
91 views

SGD versus Adam Optimization Clarification

Reading the Adam paper, I need some clarificaiton. It states that SGD optimization updates the parameters with the same learning rate (i.e. it does not change throughout training). They state Adam is ...
0
votes
0answers
14 views

When scaled conjugate gradient backpropagation is suitable for backpropagation?

I usually use gradient descent with Adam optimizer to perform backpropagation in deep learning methods. I knew it is a very efficient method. The question is in which situations we can use "scaled ...
0
votes
0answers
19 views

Save and restore custom optimizers for continued training of neural networks in TensorFlow

My question is essentially the exact same as that specified here but without using the Keras backend. Namely, how does one save and restore custom optimizers to their last state in TensorFlow (e.g. <...
0
votes
0answers
14 views

Logistic loss increasing while training with minibatches using the adam algorithm

I am trying to write my own code to use the adam algorithm for logistic regression. I am pretty sure It is training correctly as when I run it I am able to accurately classify a bunch of toy data that ...
1
vote
2answers
28 views

Appropriate loss function for multi-hot output vectors

I have some data in which model inputs and outputs (which are the same size) belong to multiple classes concurrently. A single input or output is a vector of zeros somewhere between one and four ...
1
vote
1answer
60 views

Google OR-Tools - Routing - Penalties - Adding Different Penalty to Different Location (Python)

I am using Google's OR-Tools for route optimisation. References can be found here. I am performing an optimisation where certain pick-up locations are dropped based on a penalty at each location. The ...
2
votes
2answers
60 views

Formal math notation of masked vector

I'm struggling to write my algorithm in a concise and correct way. The following is an explanation for an optimizer's update step of part of a vector of weights (not a matrix in my case). I have a ...
0
votes
0answers
6 views

Which qp solver is best for Support Vector machine implementation?

I'm trying to implement svm from scratch. I have used cvxopt to solve svm dual problem. cvxopt doesn't seem to produce accurate result when compared with other standard library. Are any other good ...
0
votes
1answer
30 views

Cat Classifier becomes worse the more you train it

I am using a dataset from kaggle to train a feed forward neural-neteork with no convolutional layers. I wanted to try it this was as a learning exercise with Pytorch without Transfer Learning and ...
0
votes
0answers
27 views

Is it ok to get different results in my implementation of svm when compared to other library?

I'm trying to implement svm for learning purpose using cvxopt in python. To check wheather my implementation works or not I compare with results of sklearn. Weights and biases of model where different ...
0
votes
0answers
9 views

Simplifying Log Likelihood equation

I am reading through a paper (https://www.mitpressjournals.org/doi/pdf/10.1162/0891201053630273) where they describe logloss as a ranking function and can be simplified to the margin of the training ...
0
votes
0answers
11 views

Best ways of encoding neural networks for GA

I am trying to create a genetic algorithm that should create optimal neural networks based on two parameters - network size and value of the fitness function, so that we can find networks that are ...

1
2 3 4 5
8