Questions tagged [optimization]

In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

Filter by
Sorted by
Tagged with
0
votes
0answers
22 views

Solve for the set of coordinates that reduces the average distances between request and server in half

I generate a DataFrame with coordinates and distances to 3 servers. ...
1
vote
0answers
33 views

Two steps optimization of a credit card limit

I have a problem similar to what is on the title but not the same, the problem on the title allows me to explain the dynamics of my need. I have to determine how much is the optimal value for a ...
0
votes
0answers
13 views

Which methods can I use to optimize my regression model so that it avoids predicting below the real value? [closed]

If I'm trying to predict something like "units sold" of a certain product to estimate the stock that I'll need. Which statistical methods or metrics can I use to train my model to avoid ...
2
votes
1answer
30 views

Adam Optimiser First Step

Plotting the paths on the cost surface from different gradient descent optimisers on a toy example, I found that the Adam algorithm does not initially travel in the direction of steepest gradient (...
0
votes
0answers
7 views

Word2Vector multicontext CBOW model with Adam optimization

In cbow multiword context word2vec model there are two weights matrixes $$ W,W^{'}$$ Where $W$ is $I$ -> $H$ weight matrix, and $W^{'}$ is $H$ -> $U$ weight matrix and output is just softmax ...
0
votes
1answer
22 views

Keras weird loss and metrics during train

I am doing some testing with tensorflow, and I bumbed into a very weird behaviour. Here is my code ...
1
vote
1answer
24 views

Find parameters to maximise output score [closed]

Not sure this is the right place to ask. Lets say there is a function f() where its implementation is unknown but it returns a score. I would like to get the ...
0
votes
0answers
11 views

How can CSO(cuckoo search optimization), PSO(particle swarm opt.) algorithms be utilized for this dataset?

How can such data be optimized using CSO, PSO algorithms so that it gives a result like which products to buy for a budget of 600$ ...
0
votes
1answer
25 views
1
vote
0answers
12 views

Optimizing a model for three different metrics that have different ranges

I have a multiple object tracker that I apply on a specific object in an image series. The tracker has several parameters that can be adjusted which affects the performance of the tracking. I am ...
0
votes
0answers
9 views

Weigthing discrimnator and generator loss in GAN networks?

Training of good generator model in vanilla GAN (Generative adversarial networks) https://papers.nips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf is achieved via minimax game, where <...
2
votes
1answer
39 views

Deep learning test loss curve won't go down

I've been working with Deep Learning projects for this current project that I am working on and it's basically a time series classification problem. Where given an array of time series data I need to ...
1
vote
0answers
9 views

Predicting quality results from operating data

Background: I have process data (table 1) that is "batch" in the chemical engineering sense of the word. Each batch ID represents the start and end of a run. Throughout the batch, different ...
1
vote
1answer
22 views

Is there a rule of thumb for a sufficient number of trials for hyperparameter search

I am implementing a quite complicated Bayesian hyperparameter search in hyperopt library on a CNN. Is there a rule of thumb for a "sufficient" number of ...
1
vote
1answer
30 views

Visualize n-dimensional bayesian optimization results

I am working on a 6-dimensional bayesian optimization problem using (skopt's gp_minimize). After the optimizer ran for j iterations I would like to somehow visualize the "progress/result" of ...
3
votes
1answer
73 views

ML/NN as Function Evaluator for further Optimization (maximization) - Practical Example

I am working on a production optimization problem; a very similar idea to what is described by Vegard Flovik How to use machine learning for production optimization. The following image, taken from ...
0
votes
1answer
15 views

Tuning the model parameters vs the parameter of optimizer for Deep Neural Networks?

I understand that there are rarely general recipes in field of machine learning and the many results can be achieved only by trial and error, and are task specific as well. My question is, if the ...
1
vote
1answer
27 views

Optimizing regression weights for NN outputs with PyTorch

So I'm basically trying to fit a regression on the relation of the input and output of a neural network model. Then the idea is, that these estimated regression weights should be optimized to some ...
0
votes
0answers
10 views

Rating Sports Teams: How to perform least squares minimization with a constraint?

I am trying to rate NFL teams by minimizing the sum of squared errors subject to a constraint. My data looks like: ...
0
votes
0answers
35 views

Levenberg–Marquardt or Adam optimization

In RBF (radial basis functions)-Neural Networks, which method (Levenberg–Marquardt or Adam optimization) is more efficient for optimizing the parameters (centers, widths, and output weights) in ...
2
votes
0answers
17 views

How to get the maximum likelihood estimate of the categorical distribution parameters using Lagrange optimization?

Let's say our data is discrete-valued and belongs to one of $K$ classes. The underlying probability distribution is assumed to be a categorical/multinoulli distribution given as $p(\textbf{x}) = \...
0
votes
0answers
37 views

Meaning of absolute tolerance and relative tolerance

Hello Data science community, I am using a library nnet in R. The documentation, page no. 4-5, shows that it has default values for ...
0
votes
0answers
6 views

How to get accurate estimates on Neural Networks Hessian?

I need to get not only accurate estimates on the neural network output itself but also on its second order derivatives in order to use the NN for optimization problems. With Adam optimizer I can't get ...
0
votes
0answers
21 views

Why is this equation converted to matrix form in this way? Is it possible to multiply an inverse matrix with a vector?

I have been banging my head on wall for days trying to decode this equation. please help me out with this... Below is the equation (consider $x$ as $\Delta x$, and $y$ as $\Delta y$): $x = - \eta(Id-\...
0
votes
1answer
33 views

Machine learning with constraints on features

I am working on a learning to rank problem. I have queries and documents related to every query which I have to rank. I used lightgbm ranker to fit the model. Some of features are very important and ...
3
votes
1answer
94 views

Comparison between cost functions to determine the “best” model?

I'm building an LSTM neural net for time series prediction (regression) and I am incorporating custom loss functions into training. I'm trying to determine which cost function (of 3 cost functions) ...
0
votes
0answers
20 views

Fast ways to multiply small matrices?

Context: I'm trying to optimize a portfolio of assets for some arbitrary goal. Currently, I'm multiplying the matrix representing the different assets with the matrix representing the weights (taken ...
0
votes
2answers
141 views

Confused between optimizer and loss function

I always thought the SGD was a loss function then I read this on a notebook ...
0
votes
2answers
40 views

Why use gradient descent on Deep Nets / RNNs when cost function is not convex?

Why do we use gradient descent on very non-convex loss functions such as in Deep nets / RNNs rather than a heuristic search (genetic algorithms, simulated annealing, etc)?
3
votes
1answer
145 views

Which learning rate should I choose?

I'm training a segmentation model, Unet++, on 2d images and I am now trying to find the optimal learning rate. The backbone of the model is Resnet34, I use Adam optimizer and the loss function is the ...
1
vote
2answers
43 views

a neural network can be used as an optimizer?

I was wondering if a neural network can be used as an optimizer. In other words, a network2 used as an optimizer takes the loss value of network1 and based on that it predicts the best weights for ...
4
votes
1answer
41 views

Need to kickstart learning rates

I was just looking at the docs on Pytorch for different available schedulers and I found one that I am having some trouble understanding here. The others seem to make sense: As training progresses, ...
0
votes
0answers
38 views

Does it make sense to use a learning rate scheduler with an adaptive learning rate method?

I'm training a large variety of networks and found out that, while using Adadelta as an optimizer, applying also the keras callback ...
1
vote
0answers
25 views

How to balance time/effort with transformations, feature selection, and models efficacy in nlp? [closed]

Edit: Question has been edited for reopening (see comment section for justification) Being to new text analytics, I haven't gotten the hang of navigating a typical workflow given the longer times ...
0
votes
0answers
25 views

optimizing customization: beyond a combinatorial approach?

Say I have a generic non-linear model (ANN, Random Forest, Gradient Boosting, etc) that wants, based on a set of features (price of a product, service duration, age, etc), to give me a prediction of ...
1
vote
1answer
30 views

Why is the exploding/vanishing gradient problem not solved by line search?

The problem of vanishing gradients is basically that since our step size is proportional to the gradient, if the gradient is very small, it might take a long time to reach a local minimum. So why don'...
0
votes
1answer
15 views

use the same gradient to maximize one part of the model and minimize another part of the same model

I want to calculate the gradient and use the same gradient to minimize one part and maximize another part of the same network (kind of adversarial case). For me, Ideal case would be, if there are two ...
0
votes
1answer
219 views

Why sparse features should have bigger learning rates associated? And how Adagrad achieves this? [closed]

I was learning about Adagrad optimizer. I came to know that it has a very helpful functionality which is that we can have lower learning rates for the features that are more common and greater ...
0
votes
1answer
57 views

Wouldn't it make more sense to give less importance to gradient far away in past in AdaGrad? [closed]

This is the update equation of a weight by AdaGrad: $$w_{new} = w_{old} - \frac{lr}{\sqrt{G_{}+E}}.G_{w_{old}}$$ Where $G$ is the sum of the gradients of the same weight at previous iterations, $E$ is ...
0
votes
1answer
98 views

Why are we taking the square root of the gradient in Adagrad? [closed]

This is how we update weights with Adagrad: $$w_i = w_i - \frac{lr}{\sqrt{g_i+E}}$$ where, $w_i$ is the $i^{th}$ weight, $lr$ is the learning rate, $g_i$ is the gradient of the $i^{th}$ weight at all ...
2
votes
4answers
187 views

Is it possible to get worse model after optimization?

I am trying recently to optimize models but for some reason, whenever I try to run the optimization the model score in the end is worse than before, so I believe I do something wrong. in order to ...
0
votes
1answer
22 views

Back Propagation Vs Learning rate in Neuralnet Optimisation

I was doing some research on how backpropagation works? I read that, backpropagation is used to find the optimal weight of each neuron after every iteration using partial derivates and updates the ...
1
vote
1answer
32 views

Good chromosome representation in a VRPTW genetic algorithm

I have a genetic algorithm for a vehicle routing problem with time windows and I need to implement certain modifications. I am not sure what would be the best chromosome representations. I have tasks ...
2
votes
1answer
191 views

When is Non-Stochastic Global Optimization Preferable or Necessary?

Background I'm specifically referring to non-convex black-box optimization problems of the form: $ \text{min} f(\vec{x})$ $s.t. \ \ a_i\le x_i \le b_i \ \forall i\in \{1,2,...,n\} \ \ \ \text{and}\ \ ...
0
votes
1answer
46 views

How To Motivate A Neural Network

Suppose a training dataset contains the following inputs: company size number of employees turnover average salary country years of operation ...and outputs: ...
2
votes
0answers
301 views

Why does Faster R-CNN use SGD optimizer instead of Adam?

I just start learning Faster R-CNN and I have some doubts about the optimizer of this network. In my understanding, Adam optimizer performs much better than SGD in a lot of networks. However, the ...
0
votes
0answers
19 views

Implementation of RMS prop for linear regression

I'm trying to implement linear regression using Rms Prop optimizer from scratch. Code: ...
3
votes
1answer
53 views

When does it make sense to choose gradient descent for SVM over liblinear?

I understand using gradient descent methods with SVM is intractable if you've used the kernel trick. In that case, best to use libsvm as your solver. But in the case that you are not using a kernel ...
4
votes
2answers
168 views

Does convergence of loss function is always guarnteed?

Which of the following is true, given the optimal learning rate? (i) For convex loss functions (i.e. with a bowl shape), batch gradient descent is guaranteed to eventually converge to the global ...
-1
votes
3answers
92 views

Why training of a neural network will require multiple iterations? [closed]

I can't understand why training of a neural network will require multiple iterations (theoretically)? Can anyone explain why, please?

1
2 3 4 5
8