# Questions tagged [optimization]

In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

499 questions
Filter by
Sorted by
Tagged with
2k views

### Setting BATCH SIZE when performing multi-class classification with imbalanced dataset

I have a question regarding BATCH_SIZE on multi-class classification task with imbalanced data. I have 5 classes and a small dataset of around ...
361 views

### Is it reasonable to train a neural network many times and cherry pick the best result based on test dataset accuracy?

My current advisor at Uni insists that I train 10 instances of the same network and pick the one with best test accuracy in order to escape the "local minima". In my opinion this does not ...
1 vote
27 views

### What is the best way to pick the optimized configuration from this dataset?

I have about 8000 configurations in an excel sheet. each configuration has four scores as seen in the image below. I would like to choose the best solution that has the highest lighting level score, ...
• 113
228 views

### Analytical solution for optimization with inequality constraints

Let the following be known matrices with dimensions as: $M = nXk$, all elements >0 $w_b = 1Xn$, all elements >0 , sums to 1 $S = nXn$, a positive semidefinite matrix $C_c = 1Xk$, all elements &...
• 131
1k views

### Scipy minimization failing with inequality constraints or bounds [closed]

I am trying to use scipy.optimize to solve a minimization problem but getting failures on using an inequality constraint or a bound. Looking for any suggestions regarding proper usage of constraints ...
• 131
388 views

### how classification scores are interpreted?

I would like to know how to interpret classification scores (i am not sure about the word score or probability, please correct me). For example, for a binary classification positive values are ...
1 vote
1k views

### How does Pytorch deal with non-differentiable activation functions during backprop?

I've read many posts on how Pytorch deal with non-differentiability in the network due to non-differentiable (or almost everywhere differentiable - doesn't make it that much better) activation ...
• 123
1k views

According to the Adam optimization update rule: $$m \leftarrow \beta_1 m + (1 - \beta_1)\nabla J(\theta)$$ $$v \leftarrow \beta_2 v + (1 - \beta_2)(\nabla J(\theta) \odot \nabla J(\theta))$$ $$\theta \... • 133 0 votes 1 answer 107 views ### Oracle in optimization I have encountered the word oracle in the following context: Given an \alpha-approximate oracle for stochastic optimization we show how to implement an \alpha-approximate solution for robust ... • 129 6 votes 2 answers 7k views ### Difference between RMSProp and Momentum? Can someone please tell me the clear difference between the approaches of RMSProp and Gradient Descent with Momentum ? Both try to achieve the same effect . One of the blogs that I read states the ... • 277 2 votes 1 answer 566 views ### Reversing the input and output of an ML algorithm to Optimize My dataset consists of multiple input variables (X) and multiple output variables (Y). For example: ... 4 votes 1 answer 3k views ### SGD versus Adam Optimization Clarification Reading the Adam paper, I need some clarificaiton. It states that SGD optimization updates the parameters with the same learning rate (i.e. it does not change throughout training). They state Adam is ... • 419 0 votes 0 answers 77 views ### Logistic loss increasing while training with minibatches using the adam algorithm I am trying to write my own code to use the adam algorithm for logistic regression. I am pretty sure It is training correctly as when I run it I am able to accurately classify a bunch of toy data that ... 2 votes 2 answers 1k views ### Appropriate loss function for multi-hot output vectors I have some data in which model inputs and outputs (which are the same size) belong to multiple classes concurrently. A single input or output is a vector of zeros somewhere between one and four ... • 194 1 vote 1 answer 715 views ### Google OR-Tools - Routing - Penalties - Adding Different Penalty to Different Location (Python) I am using Google's OR-Tools for route optimisation. References can be found here. I am performing an optimisation where certain pick-up locations are dropped based on a penalty at each location. The ... • 1,615 2 votes 2 answers 518 views ### Formal math notation of masked vector I'm struggling to write my algorithm in a concise and correct way. The following is an explanation for an optimizer's update step of part of a vector of weights (not a matrix in my case). I have a ... • 145 0 votes 1 answer 43 views ### Cat Classifier becomes worse the more you train it I am using a dataset from kaggle to train a feed forward neural-neteork with no convolutional layers. I wanted to try it this was as a learning exercise with Pytorch without Transfer Learning and ... 0 votes 1 answer 838 views ### Hill Climbing Algorithm - Optimum Step Size I am implementing a standard hill climbing algorithm to optimise hyper-parameters for a predictive model. The hill climbing algorithm is being applied as part of a two-stage approach: Apply grid ... • 30 1 vote 2 answers 826 views ### How to tell my neural network that I care much more about precision than recall? I am training a neural network for a multilabel classification problem, so my last layer consist of n_classes sigmoid neurons. Now, I know that it is impossible to ... • 1,170 0 votes 1 answer 29 views ### Optimization function returns the same optimal parameters for two labels I've recently enrolled in the Coursera machine learning, and am working my way through making my own classifier for the Iris dataset problem using matlab. I'm training a classifier for each species (... 0 votes 1 answer 458 views ### What is the objective that is optimized with Random Search? I have recently learned about Random Search (or sklearn.model_selection.RandomizedSearchCV in Python) and was thinking about the theory behind the optimization process. In particular my question is, ... 1 vote 1 answer 785 views ### how to compute bernoulli entropy? I am reading gail implementation code in openai baselines. they compute bernoulli entropy as one of the loss in adversary network loss function. In their code, they implement bernoulli entropy as ... 1 vote 0 answers 55 views ### Negative impact of "important" features on model performance I have a random forest regressor with a set of base features, fit & optimised with sklearn random search algorithm. When I add a set of additional features and retrain (again with random search ... • 21 3 votes 0 answers 96 views ### Dissecting and understanding the Adam optimization's formula Adam's optimization has the fololwing parameter update rule :$$ \theta_{t+1} = \theta_{t} - \alpha*\dfrac{m_t}{\sqrt{v_t + \epsilon}}$$where$$ m_t \text{ is first moment of gradients and} \space ...
631 views

The following paper ADADELTA: AN ADAPTIVE LEARNING RATE METHOD gives a method called Adagrad where we we have the following update rule : $$X_{n+1} = X_n -[Lr/\sqrt{\sum_{i=0}^ng_i^2}]*g_n$$ Now I ...
1 vote
243 views

### Does optimizer highly affect on accuracy?

I used SGD as optimizer and its accuracy result is about 97% and I have changed optimizer to Adam surprisingly, my accuracy became 49% I only changed optimizer and didn`t change anything else but ...
1 vote
137 views

### Finding a vector that minimize the MSE of its linear combination

I have been doing a COVID-19 related project. Here is the question: N = vector of daily new infected cases D = vector of daily deaths E[D] = estimation of daily deaths N is a n-dimensional vector, n ...
1 vote
22 views

### Optimization problem with different type of constraints

I'm new to optimization problems. I want to find optimum values for my objective function. You can imagine my function as E = f(t1, t2, t3). I want to minimize <...
• 121
1 vote
161 views

### Best approach to find optimal solution to linear equation by group in R

I am currently modeling a pricing and discount system in R. My data frame looks as follows: ...
• 1,743
2k views

### Optimising for Brier objective function directly gives worse Brier score than optimising with custom objective - what does it tell me?

I am training an XGBoost model and as I care the most about resulting probabilities, not classification itself I have chosen Brier score as a metric for my model, so that probabilities would be well ...
• 192
1 vote
12 views

### Basket items optimisation minimising constraints

I have a real problem (not home work) when I have to distribute an ordered list by position to respect some constraints eg. 1. 11 2. 15 3. 18 4. 18 5. 1 baskets:...
• 111
1 vote
475 views

### Computing variance of an SGD iteration

It is known that SGD iteration has huge variance. Given the iteration update: $$w^{k+1} := w^k - \underbrace{\alpha \ g_i(w^k)}_{p^k},$$ where $w$ are model weights and $g_i(w^k)$ is gradient of ...
1 vote
27 views

### Is Neural Network Architecture independent of Data?

If I change my dataset (let's say it is always images), should I change the architecture of my neural network?
• 143
1 vote
244 views

### Does severe multicollinearity affect solving linear regression by gradient descent?

Since OLS may fail when there is severe/near perfect multicollinearity, how would gradient descent perform in such a scenario? Does it converge at the minima? (My guess is, Cost function of linear ...
• 195
1 vote
96 views

### How does the construction of a decision tree differ for different optimization metrics?

I understand how a decision tree is constructed (in the ID3 algorithm) using criterion such as entropy, gini index, and variance reduction. But the formulae for these criteria do not care about ...
• 111
1 vote
17 views

### Need ideas on how to find the best email send frequency go get maximum desired action

I have got data which contains email id total times email has been send to him (frequerny) and desired action taken (read, clicked etc), non desired action taken(unsubscribed, etc), also count of no ...
• 141
11k views

### How to Minimize mean square error using Python

I want to minimise mean square error function to find best alpha value (decay rate) for my model. Here is the description of my model: ...
• 277
1 vote
2k views

### Differential Evolution optimal tolerance parameter

I am trying to optimize the parameters of a global optimization system for my set of data, because I will have a bunch of similar data to process so I need to fine tune the global optimizator so that ...
2k views

### How many times is backprop used in epoch?

As I understand for the algorithms that use gradient descent we have to pass data to the algorithms multiple times so that the optimum is found. So one epoch means that the forward-backprop (and ...
• 153
84 views

### Solve an equation using machine learning [closed]

Imagine we have the following equation: y=xz. We have y but not other ones. Note that y is like a matrix and we could as many sample we want. It is the values obtained from sensors. This means it ...
• 453
1 vote
115 views

### Reinforcement Learning : Why acting greedily with the optimal value function gives you the optimal policy?

The course of David Silver about Reinforcement Learning explains how you get the optimal policy from the optimal value function. It seems to be very simple, you just have to act greedily, by ...
• 11
1 vote
140 views

### ADAM algorithm for multilayer neural network

I’m trying to touch neural networks without using “in box” algorithms. And so I found out that nowhere is written how to calculate square of gradient for hidden layers in ADAM optimizer. I took the ...
65 views

### How to optimize client's portafolio with analytical models?

I have a model in which we want to optimize the probability of an outcome depending on a election of some product (a personalized product for every client amongst three posibilities). The product is ...
1 vote
116 views

### Can we optimize heterogeneous parameters of RBF Network using Gradient Descent?

There're three parameters in the Radial Basis Function Networks (RBFN). Centers of RBFs Width of RBFs Weights of RBFs It's a fact that Weights can be easily updated using a simple Gradient Descent. ...
• 271
72 views

### Salesman problem with additional conditions and features [closed]

I'd like to specify the kind of a problem I encountered. I need to compose the best route for a car driver who goes to different cities. His aim to check in the most proper car park in a city taking ...
• 301
256 views

### Constructing function - f(x,y) for the given minimums (Python) [closed]

Problem Statement: I need to construct a function f(x,y) in which there're 3 minimums. 2 local and 1 global which are written below. Locals are: z = f(0.2,0.3) = 0.7 | z = f(0.6,0.8) = 0.8 Global is: ...
• 271
193 views

### What applications does linear programming have in data science?

I'm currently learning about linear programming in my degree. I'm wondering how this is relevant to anything in data science?
• 467
106 views

### Can I completely cancel the effects of using a smaller batch size by reducing the learning rate?

I'm having the problem that the data from a regular sized batch (e.g., 32, 64) doesn't fit in my GPU. Among other solutions, I'm considering reducing the batch size, as is normally suggested. Of ...
5k views

### Is it possible to make F1_Score differentiable and use it directly as a Loss function?

One of the metrics that is widely used in binary classification is the F1 score: $F_1 = 2\cdot \frac{recall \cdot precision}{recall+precision}$ The problem of the F1-score is that it is not ...
• 6,282
1 vote