Skip to main content

Questions tagged [optimization]

In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

Filter by
Sorted by
Tagged with
1 vote
0 answers
96 views

How to maximize a log linear regression equation satisfying a constraint?

I have a log linear equation of the form $y = w_1(\log{X1}) + w_2(\log{X2}) + ... + w_n(\log{Xn})$. How can I find the value of X's that maximize the value of y subject to a constraint $(X_1+X_2+...+...
my_cse lab's user avatar
2 votes
1 answer
173 views

What method/algorithm for constrained multi-target regression

I am working with three dimensional measurement data and want to model them using a multivariate linear regression. I have already implemented a simple gradient descent algorithm to solve the classic ...
schafran's user avatar
  • 131
1 vote
0 answers
30 views

Finding the dual to an optimization problem on an unsupervised dataset [closed]

We consider the unsupervised dataset $x_1,..x_N \in R^d$ and the optimization problem: $$min_w \,\frac{1}{2}{\left\lVert w \right\rVert}^2,$$ subject to constraints:$$\forall_{i=1}^N: \phi(x_i)^Tw\...
user113198's user avatar
1 vote
1 answer
157 views

Terminology to distinguish between ML methods and optimization methods (PSO, ACO..)

I am currently writing a scientific thesis which consists of two parts. In the first part I am building ML models with neural networks, support vectors etc. and the second part is about finding global ...
Emma's user avatar
  • 65
1 vote
0 answers
54 views

Optimize Yahoo Finance Code for Analysis [closed]

I am trying to analyze a number of companies using financial data I gathered from Yahoo Finance. I am also using the yfinance API to get some more details about the company using functions. Since I am ...
m2rik's user avatar
  • 321
0 votes
1 answer
124 views

Gradient descent around optimal loss surface

All the loss surface used in examples have some of bowl shape that decrease drastically far from the optimal and decrease slowly around the optimal flat point. My questions are: Has all the loss ...
jpark's user avatar
  • 1
3 votes
1 answer
544 views

Two steps optimization of a credit card limit

I have a problem similar to what is on the title but not the same. The problem on the title allows me to explain the dynamics of my need. I have to determine what the optimal value is for a variable ...
Juan Esteban de la Calle's user avatar
2 votes
1 answer
273 views

Adam Optimiser First Step

Plotting the paths on the cost surface from different gradient descent optimisers on a toy example, I found that the Adam algorithm does not initially travel in the direction of steepest gradient (...
foam78's user avatar
  • 123
0 votes
1 answer
264 views

Keras weird loss and metrics during train

I am doing some testing with tensorflow, and I bumbed into a very weird behaviour. Here is my code ...
Dave's user avatar
  • 13
1 vote
1 answer
469 views

Find parameters to maximise output score [closed]

Not sure this is the right place to ask. Lets say there is a function f() where its implementation is unknown but it returns a score. I would like to get the ...
Chris's user avatar
  • 155
0 votes
1 answer
59 views

What is "SwarmPackagePy.cso.cso at 0x187cf21e340"

...
ÖzgürP's user avatar
  • 111
1 vote
0 answers
17 views

Optimizing a model for three different metrics that have different ranges

I have a multiple object tracker that I apply on a specific object in an image series. The tracker has several parameters that can be adjusted which affects the performance of the tracking. I am ...
Hjalte's user avatar
  • 111
2 votes
1 answer
697 views

Deep learning test loss curve won't go down

I've been working with Deep Learning projects for this current project that I am working on and it's basically a time series classification problem. Where given an array of time series data I need to ...
Joseph Anderson's user avatar
1 vote
0 answers
20 views

Predicting quality results from operating data

Background: I have process data (table 1) that is "batch" in the chemical engineering sense of the word. Each batch ID represents the start and end of a run. Throughout the batch, different ...
user110294's user avatar
3 votes
1 answer
1k views

Is there a rule of thumb for a sufficient number of trials for hyperparameter search

I am implementing a quite complicated Bayesian hyperparameter search in hyperopt library on a CNN. Is there a rule of thumb for a "sufficient" number of ...
Leevo's user avatar
  • 6,265
1 vote
1 answer
261 views

Visualize n-dimensional bayesian optimization results

I am working on a 6-dimensional bayesian optimization problem using (skopt's gp_minimize). After the optimizer ran for j iterations I would like to somehow visualize the "progress/result" of ...
FR_MPI's user avatar
  • 11
4 votes
2 answers
186 views

ML/NN as Function Evaluator for further Optimization (maximization) - Practical Example

I am working on a production optimization problem; a very similar idea to what is described by Vegard Flovik How to use machine learning for production optimization. The following image, taken from ...
TwinPenguins's user avatar
  • 4,279
0 votes
1 answer
46 views

Tuning the model parameters vs the parameter of optimizer for Deep Neural Networks?

I understand that there are rarely general recipes in field of machine learning and the many results can be achieved only by trial and error, and are task specific as well. My question is, if the ...
spiridon_the_sun_rotator's user avatar
1 vote
1 answer
124 views

Optimizing regression weights for NN outputs with PyTorch

So I'm basically trying to fit a regression on the relation of the input and output of a neural network model. Then the idea is, that these estimated regression weights should be optimized to some ...
T.Tos's user avatar
  • 41
2 votes
0 answers
251 views

How to get the maximum likelihood estimate of the categorical distribution parameters using Lagrange optimization?

Let's say our data is discrete-valued and belongs to one of $K$ classes. The underlying probability distribution is assumed to be a categorical/multinoulli distribution given as $p(\textbf{x}) = \...
Shashank Kumar's user avatar
0 votes
1 answer
145 views

Machine learning with constraints on features

I am working on a learning to rank problem. I have queries and documents related to every query which I have to rank. I used lightgbm ranker to fit the model. Some of features are very important and ...
Siddhant Tandon's user avatar
3 votes
1 answer
195 views

Comparison between cost functions to determine the "best" model?

I'm building an LSTM neural net for time series prediction (regression) and I am incorporating custom loss functions into training. I'm trying to determine which cost function (of 3 cost functions) ...
PyRsquared's user avatar
  • 1,614
0 votes
0 answers
93 views

Fast ways to multiply small matrices?

Context: I'm trying to optimize a portfolio of assets for some arbitrary goal. Currently, I'm multiplying the matrix representing the different assets with the matrix representing the weights (taken ...
pythonbygone's user avatar
1 vote
2 answers
6k views

Confused between optimizer and loss function

I always thought the SGD was a loss function then I read this on a notebook ...
Hanna polaskus's user avatar
1 vote
2 answers
64 views

Why use gradient descent on Deep Nets / RNNs when cost function is not convex?

Why do we use gradient descent on very non-convex loss functions such as in Deep nets / RNNs rather than a heuristic search (genetic algorithms, simulated annealing, etc)?
user2351494's user avatar
3 votes
1 answer
1k views

Which learning rate should I choose?

I'm training a segmentation model, Unet++, on 2d images and I am now trying to find the optimal learning rate. The backbone of the model is Resnet34, I use Adam optimizer and the loss function is the ...
Nicolas's user avatar
  • 244
2 votes
2 answers
76 views

a neural network can be used as an optimizer?

I was wondering if a neural network can be used as an optimizer. In other words, a network2 used as an optimizer takes the loss value of network1 and based on that it predicts the best weights for ...
molo32's user avatar
  • 179
4 votes
1 answer
53 views

Need to kickstart learning rates

I was just looking at the docs on Pytorch for different available schedulers and I found one that I am having some trouble understanding here. The others seem to make sense: As training progresses, ...
Pawan Bhandarkar's user avatar
1 vote
0 answers
25 views

How to balance time/effort with transformations, feature selection, and models efficacy in nlp? [closed]

Edit: Question has been edited for reopening (see comment section for justification) Being to new text analytics, I haven't gotten the hang of navigating a typical workflow given the longer times ...
Josh's user avatar
  • 141
1 vote
1 answer
99 views

Why is the exploding/vanishing gradient problem not solved by line search?

The problem of vanishing gradients is basically that since our step size is proportional to the gradient, if the gradient is very small, it might take a long time to reach a local minimum. So why don'...
Jack M's user avatar
  • 265
0 votes
1 answer
335 views

use the same gradient to maximize one part of the model and minimize another part of the same model

I want to calculate the gradient and use the same gradient to minimize one part and maximize another part of the same network (kind of adversarial case). For me, Ideal case would be, if there are two ...
user3363813's user avatar
0 votes
1 answer
2k views

Why sparse features should have bigger learning rates associated? And how Adagrad achieves this? [closed]

I was learning about Adagrad optimizer. I came to know that it has a very helpful functionality which is that we can have lower learning rates for the features that are more common and greater ...
user avatar
0 votes
1 answer
172 views

Wouldn't it make more sense to give less importance to gradient far away in past in AdaGrad? [closed]

This is the update equation of a weight by AdaGrad: $$w_{new} = w_{old} - \frac{lr}{\sqrt{G_{}+E}}.G_{w_{old}}$$ Where $G$ is the sum of the gradients of the same weight at previous iterations, $E$ is ...
Dhruv Agarwal's user avatar
1 vote
1 answer
686 views

Why are we taking the square root of the gradient in Adagrad? [closed]

This is how we update weights with Adagrad: $$w_i = w_i - \frac{lr}{\sqrt{g_i+E}}$$ where, $w_i$ is the $i^{th}$ weight, $lr$ is the learning rate, $g_i$ is the gradient of the $i^{th}$ weight at all ...
Dhruv Agarwal's user avatar
3 votes
4 answers
5k views

Is it possible to get worse model after optimization?

I am trying recently to optimize models but for some reason, whenever I try to run the optimization the model score in the end is worse than before, so I believe I do something wrong. in order to ...
Reut's user avatar
  • 349
0 votes
1 answer
40 views

Back Propagation Vs Learning rate in Neuralnet Optimisation

I was doing some research on how backpropagation works? I read that, backpropagation is used to find the optimal weight of each neuron after every iteration using partial derivates and updates the ...
Jack Daniel's user avatar
1 vote
1 answer
99 views

Good chromosome representation in a VRPTW genetic algorithm

I have a genetic algorithm for a vehicle routing problem with time windows and I need to implement certain modifications. I am not sure what would be the best chromosome representations. I have tasks ...
Ivva Hamerníková's user avatar
2 votes
1 answer
156 views

Why do we only care about convex functions when doing Gradient Descent/SGD?

I mean I know why we specifically care about convex functions: it's because their local minimum are also global, and so you just have to "follow a path which goes down" to find the minima of ...
lairv's user avatar
  • 39
2 votes
1 answer
352 views

When is Non-Stochastic Global Optimization Preferable or Necessary?

Background I'm specifically referring to non-convex black-box optimization problems of the form: $ \text{min} f(\vec{x})$ $s.t. \ \ a_i\le x_i \le b_i \ \forall i\in \{1,2,...,n\} \ \ \ \text{and}\ \ ...
Ben's user avatar
  • 2,572
0 votes
1 answer
60 views

How To Motivate A Neural Network

Suppose a training dataset contains the following inputs: company size number of employees turnover average salary country years of operation ...and outputs: ...
Alan's user avatar
  • 111
2 votes
1 answer
3k views

Why does Faster R-CNN use SGD optimizer instead of Adam?

I just start learning Faster R-CNN and I have some doubts about the optimizer of this network. In my understanding, Adam optimizer performs much better than SGD in a lot of networks. However, the ...
icebear's user avatar
  • 21
1 vote
0 answers
268 views

Implementation of RMS prop for linear regression

I'm trying to implement linear regression using Rms Prop optimizer from scratch. Code: ...
Aniket Bote's user avatar
3 votes
1 answer
226 views

When does it make sense to choose gradient descent for SVM over liblinear?

I understand using gradient descent methods with SVM is intractable if you've used the kernel trick. In that case, best to use libsvm as your solver. But in the case that you are not using a kernel ...
Learning stats by example's user avatar
4 votes
2 answers
2k views

Does convergence of loss function is always guarnteed?

Which of the following is true, given the optimal learning rate? (i) For convex loss functions (i.e. with a bowl shape), batch gradient descent is guaranteed to eventually converge to the global ...
MAC's user avatar
  • 277
-1 votes
3 answers
472 views

Why training of a neural network will require multiple iterations? [closed]

I can't understand why training of a neural network will require multiple iterations (theoretically)? Can anyone explain why, please?
Ali's user avatar
  • 17
0 votes
1 answer
117 views

"Invalid value" in RMSprop implementation from scratch in Python

Edit 2: The regularization term (reg_term) is sometimes negative due negatative parameters. Hence S[f"dW{l}"] contains some negative values. I realize the reg_term has to be added before ...
Simen's user avatar
  • 1
3 votes
1 answer
320 views

Support Vector Machines with soft margin: solving the dual form

I am currently struggling with finding an analytical solution for the $\alpha_k$. I have derived the following constrained optimization problem: $$ L = \sum_{i=1}^{N} \alpha_i - \frac{1}{2} \sum_{i=1}^...
Miistogun's user avatar
1 vote
0 answers
37 views

Using Iterative Hard/Soft Thresholding in autoencoder with non linear activation

Can someone please give an intuitive explanation of the difference between the Iterative Hard Thresholding VS Iterative Soft thresholding algorithm? And if we can use these algorithms in an ...
Maria's user avatar
  • 331
2 votes
1 answer
197 views

how to find the best parameters to solve a differential equation? [closed]

I have a differential equation: def func(Y, t, r, p, K, alpha): return r * (Y ** p) * (1 - (Y / K) ** alpha) and I want to find the best parameters that fit (r,...
Hassan's user avatar
  • 21
2 votes
1 answer
2k views

Setting BATCH SIZE when performing multi-class classification with imbalanced dataset

I have a question regarding BATCH_SIZE on multi-class classification task with imbalanced data. I have 5 classes and a small dataset of around ...
Stefan Radonjic's user avatar

1 2 3
4
5
10