Questions tagged [optimization]

In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

Filter by
Sorted by
Tagged with
1
vote
1answer
13 views

how to find the best parameters to solve a differential equation?

I have a differential equation: def func(Y, t, r, p, K, alpha): return r * (Y ** p) * (1 - (Y / K) ** alpha) and I want to find the best parameters that fit (r,...
1
vote
0answers
13 views

Setting BATCH SIZE when performing multi-class classification with imbalanced dataset

I have a question regarding BATCH_SIZE on multi-class classification task with imbalanced data. I have 5 classes and a small dataset of around ...
2
votes
2answers
36 views

Is it reasonable to train a neural network many times and cherry pick the best result based on test dataset accuracy?

My current advisor at Uni insists that I train 10 instances of the same network and pick the one with best test accuracy in order to escape the "local minima". In my opinion this does not ...
0
votes
1answer
16 views

What is the best way to pick the optimized configuration from this dataset?

I have about 8000 configurations in an excel sheet. each configuration has four scores as seen in the image below. I would like to choose the best solution that has the highest lighting level score, ...
0
votes
0answers
17 views

Analytical solution for optimization with inequality constraints

Let the following be known matrices with dimensions as: $M = nXk$, all elements >0 $w_b = 1Xn$, all elements >0 , sums to 1 $S = nXn$, a positive semidefinite matrix $C_c = 1Xk$, all elements &...
3
votes
0answers
25 views

Scipy minimization failing with inequality constraints or bounds

I am trying to use scipy.optimize to solve a minimization problem but getting failures on using an inequality constraint or a bound. Looking for any suggestions regarding proper usage of constraints ...
-1
votes
0answers
5 views

Writing price optimization program in PuLP

I am trying to write a small price optimization engine that optimizes revenues given a list of articles. I have a list of articles and for each of them, I have its price elasticity of demand. My ...
0
votes
1answer
17 views

How to improve pandas DataFrame lookup [closed]

Need help to optimize this code, just for 256553 records and about 100 MB DataFrame size, this code takes 25-30 minute to execute. Basically what I am doing here ...
2
votes
2answers
25 views

how classification scores are interpreted?

I would like to know how to interpret classification scores (i am not sure about the word score or probability, please correct me). For example, for a binary classification positive values are ...
0
votes
1answer
15 views

How does Pytorch deal with non-differentiable activation functions during backprop?

I've read many posts on how Pytorch deal with non-differentiability in the network due to non-differentiable (or almost everywhere differentiable - doesn't make it that much better) activation ...
1
vote
1answer
24 views

How similar is Adam optimization and Gradient clipping?

According to the Adam optimization update rule: $$m \leftarrow \beta_1 m + (1 - \beta_1)\nabla J(\theta)$$ $$v \leftarrow \beta_2 v + (1 - \beta_2)(\nabla J(\theta) \odot \nabla J(\theta))$$ $$\theta \...
0
votes
1answer
50 views

Oracle in optimization

I have encountered the word oracle in the following context: Given an $\alpha$-approximate oracle for stochastic optimization we show how to implement an $\alpha$-approximate solution for robust ...
3
votes
2answers
73 views

Difference between RMSProp and Momentum?

Can someone please tell me the clear difference between the approaches of RMSProp and Gradient Descent with Momentum ? Both try to achieve the same effect . One of the blogs that I read states the ...
1
vote
1answer
19 views

Reversing the input and output of an ML algorithm to Optimize

My dataset consists of multiple input variables (X) and multiple output variables (Y). For example: ...
3
votes
0answers
40 views

SGD versus Adam Optimization Clarification

Reading the Adam paper, I need some clarificaiton. It states that SGD optimization updates the parameters with the same learning rate (i.e. it does not change throughout training). They state Adam is ...
0
votes
0answers
10 views

When scaled conjugate gradient backpropagation is suitable for backpropagation?

I usually use gradient descent with Adam optimizer to perform backpropagation in deep learning methods. I knew it is a very efficient method. The question is in which situations we can use "scaled ...
0
votes
0answers
5 views

Save and restore custom optimizers for continued training of neural networks in TensorFlow

My question is essentially the exact same as that specified here but without using the Keras backend. Namely, how does one save and restore custom optimizers to their last state in TensorFlow (e.g. <...
0
votes
0answers
11 views

Logistic loss increasing while training with minibatches using the adam algorithm

I am trying to write my own code to use the adam algorithm for logistic regression. I am pretty sure It is training correctly as when I run it I am able to accurately classify a bunch of toy data that ...
1
vote
2answers
23 views

Appropriate loss function for multi-hot output vectors

I have some data in which model inputs and outputs (which are the same size) belong to multiple classes concurrently. A single input or output is a vector of zeros somewhere between one and four ...
1
vote
1answer
35 views

Google OR-Tools - Routing - Penalties - Adding Different Penalty to Different Location (Python)

I am using Google's OR-Tools for route optimisation. References can be found here. I am performing an optimisation where certain pick-up locations are dropped based on a penalty at each location. The ...
2
votes
2answers
42 views

Formal math notation of masked vector

I'm struggling to write my algorithm in a concise and correct way. The following is an explanation for an optimizer's update step of part of a vector of weights (not a matrix in my case). I have a ...
0
votes
0answers
5 views

Which qp solver is best for Support Vector machine implementation?

I'm trying to implement svm from scratch. I have used cvxopt to solve svm dual problem. cvxopt doesn't seem to produce accurate result when compared with other standard library. Are any other good ...
0
votes
1answer
27 views

Cat Classifier becomes worse the more you train it

I am using a dataset from kaggle to train a feed forward neural-neteork with no convolutional layers. I wanted to try it this was as a learning exercise with Pytorch without Transfer Learning and ...
0
votes
0answers
26 views

Is it ok to get different results in my implementation of svm when compared to other library?

I'm trying to implement svm for learning purpose using cvxopt in python. To check wheather my implementation works or not I compare with results of sklearn. Weights and biases of model where different ...
0
votes
0answers
9 views

Simplifying Log Likelihood equation

I am reading through a paper (https://www.mitpressjournals.org/doi/pdf/10.1162/0891201053630273) where they describe logloss as a ranking function and can be simplified to the margin of the training ...
0
votes
0answers
11 views

Best ways of encoding neural networks for GA

I am trying to create a genetic algorithm that should create optimal neural networks based on two parameters - network size and value of the fitness function, so that we can find networks that are ...
0
votes
0answers
30 views

How to compute error (instead of gradient of error) for each node in backpropagation?

I was going through the relevant chain rule mathematics and I have successfully implemented backpropagation from scratch for MNIST (once, I even tried doing this for a small sample data I created by ...
0
votes
1answer
59 views

Hill Climbing Algorithm - Optimum Step Size

I am implementing a standard hill climbing algorithm to optimise hyper-parameters for a predictive model. The hill climbing algorithm is being applied as part of a two-stage approach: Apply grid ...
0
votes
0answers
8 views

How to design a two-factor optimization score function?

What if I have two parameters as optimization target and want to maximize one and minimize the other one? Example: food products dataset, look for strawberries and artichokes in USDA FDC. Task: find ...
1
vote
2answers
41 views

How to tell my neural network that I care much more about precision than recall?

I am training a neural network for a multilabel classification problem, so my last layer consist of n_classes sigmoid neurons. Now, I know that it is impossible to ...
0
votes
0answers
23 views

Am I doing bayesian optimization correctly for MLP?

I am trying to optimize the below mentioned hyper-parameters of MLP with range as follows. Number of hidden layers (n): 1-10 Number of perceptrons (p): 25, 50, 75, ..., 200 Activation function: ...
0
votes
1answer
13 views

Optimization function returns the same optimal parameters for two labels

I've recently enrolled in the Coursera machine learning, and am working my way through making my own classifier for the Iris dataset problem using matlab. I'm training a classifier for each species (...
0
votes
1answer
22 views

What is the objective that is optimized with Random Search?

I have recently learned about Random Search (or sklearn.model_selection.RandomizedSearchCV in Python) and was thinking about the theory behind the optimization process. In particular my question is, ...
0
votes
1answer
35 views

how to compute bernoulli entropy?

I am reading gail implementation code in openai baselines. they compute bernoulli entropy as one of the loss in adversary network loss function. In their code, they implement bernoulli entropy as ...
0
votes
0answers
13 views

Convergence check of constrained biconvex optimization problem

I participate in the development of a matrix factorization algorithm and I have some convergence issues. Here is the kind of minimization problem I am facing : $L(D,A) = ||X-DA||^2 + \sum_{(i,j) \in ...
0
votes
0answers
5 views

Reward engineering to replace single terminal reward (exponential utility of terminal wealth)

My goal is to use reinforcement learning to train the agent (the trader) to maximize the exponential utility of his P&L (profit and loss) at a terminal time T. Therefore the natural formulation of ...
0
votes
0answers
11 views

Negative impact of “important” features on model performance

I have a random forest regressor with a set of base features, fit & optimised with sklearn random search algorithm. When I add a set of additional features and retrain (again with random search ...
2
votes
0answers
30 views

Dissecting and understanding the Adam optimization's formula

Adam's optimization has the fololwing parameter update rule : $$ \theta_{t+1} = \theta_{t} - \alpha*\dfrac{m_t}{\sqrt{v_t + \epsilon}}$$ where $$ m_t \text{ is first moment of gradients and} \space ...
2
votes
1answer
37 views

Intuition behind Adagrad optimization

The following paper ADADELTA: AN ADAPTIVE LEARNING RATE METHOD gives a method called Adagrad where we we have the following update rule : $$ X_{n+1} = X_n -[Lr/\sqrt{\sum_{i=0}^ng_i^2}]*g_n $$ Now I ...
1
vote
0answers
15 views

Does optimizer highly affect on accuracy?

I used SGD as optimizer and its accuracy result is about 97% and I have changed optimizer to Adam surprisingly, my accuracy became 49% I only changed optimizer and didn`t change anything else but ...
0
votes
1answer
24 views

Finding a vector that minimize the MSE of its linear combination

I have been doing a COVID-19 related project. Here is the question: N = vector of daily new infected cases D = vector of daily deaths E[D] = estimation of daily deaths N is a n-dimensional vector, n ...
1
vote
0answers
17 views

Optimization problem with different type of constraints

I'm new to optimization problems. I want to find optimum values for my objective function. You can imagine my function as E = f(t1, t2, t3). I want to minimize <...
1
vote
1answer
85 views

Best approach to find optimal solution to linear equation by group in R

I am currently modeling a pricing and discount system in R. My data frame looks as follows: ...
5
votes
2answers
117 views

Optimising for Brier objective function directly gives worse Brier score than optimising with custom objective - what does it tell me?

I am training an XGBoost model and as I care the most about resulting probabilities, not classification itself I have chosen Brier score as a metric for my model, so that probabilities would be well ...
0
votes
0answers
10 views

Basket items optimisation minimising constraints

I have a real problem (not home work) when I have to distribute an ordered list by position to respect some constraints eg. 1. 11 2. 15 3. 18 4. 18 5. 1 baskets:...
1
vote
1answer
56 views

Computing variance of an SGD iteration

It is known that SGD iteration has huge variance. Given the iteration update: $$ w^{k+1} := w^k - \underbrace{\alpha \ g_i(w^k)}_{p^k}, $$ where $w$ are model weights and $g_i(w^k)$ is gradient of ...
1
vote
1answer
23 views

Is Neural Network Architecture independent of Data?

If I change my dataset (let's say it is always images), should I change the architecture of my neural network?
0
votes
0answers
7 views

what does “Tree” refer to in Tree-structured Parzen Estimators

I am going through the literature of Hyperparameter optimization techniques and came across TPE. There is very little to no explanation on why the name has "Tree" in it. What is Tree referring to? and ...
0
votes
0answers
11 views

Does severe multicollinearity affect solving linear regression by gradient descent?

Since OLS may fail when there is severe/near perfect multicollinearity, how would gradient descent perform in such a scenario? Does it converge at the minima? (My guess is, Cost function of linear ...
1
vote
1answer
24 views

How does the construction of a decision tree differ for different optimization metrics?

I understand how a decision tree is constructed (in the ID3 algorithm) using criterion like entropy, gini index, variance reduction. But the formulae for these criteria do not care about optimization ...

1
2 3 4 5
7