Skip to main content

Questions tagged [optimization]

In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

Filter by
Sorted by
Tagged with
2 votes
1 answer
2k views

Setting BATCH SIZE when performing multi-class classification with imbalanced dataset

I have a question regarding BATCH_SIZE on multi-class classification task with imbalanced data. I have 5 classes and a small dataset of around ...
Stefan Radonjic's user avatar
2 votes
2 answers
361 views

Is it reasonable to train a neural network many times and cherry pick the best result based on test dataset accuracy?

My current advisor at Uni insists that I train 10 instances of the same network and pick the one with best test accuracy in order to escape the "local minima". In my opinion this does not ...
Heitor da Rocha Coimbra's user avatar
1 vote
1 answer
27 views

What is the best way to pick the optimized configuration from this dataset?

I have about 8000 configurations in an excel sheet. each configuration has four scores as seen in the image below. I would like to choose the best solution that has the highest lighting level score, ...
Julia_arch's user avatar
0 votes
0 answers
228 views

Analytical solution for optimization with inequality constraints

Let the following be known matrices with dimensions as: $M = nXk$, all elements >0 $w_b = 1Xn$, all elements >0 , sums to 1 $S = nXn$, a positive semidefinite matrix $C_c = 1Xk$, all elements &...
dayum's user avatar
  • 131
3 votes
0 answers
1k views

Scipy minimization failing with inequality constraints or bounds [closed]

I am trying to use scipy.optimize to solve a minimization problem but getting failures on using an inequality constraint or a bound. Looking for any suggestions regarding proper usage of constraints ...
dayum's user avatar
  • 131
2 votes
2 answers
388 views

how classification scores are interpreted?

I would like to know how to interpret classification scores (i am not sure about the word score or probability, please correct me). For example, for a binary classification positive values are ...
phillipe cauchett's user avatar
1 vote
1 answer
1k views

How does Pytorch deal with non-differentiable activation functions during backprop?

I've read many posts on how Pytorch deal with non-differentiability in the network due to non-differentiable (or almost everywhere differentiable - doesn't make it that much better) activation ...
Norman's user avatar
  • 123
3 votes
2 answers
1k views

How similar is Adam optimization and Gradient clipping?

According to the Adam optimization update rule: $$m \leftarrow \beta_1 m + (1 - \beta_1)\nabla J(\theta)$$ $$v \leftarrow \beta_2 v + (1 - \beta_2)(\nabla J(\theta) \odot \nabla J(\theta))$$ $$\theta \...
Shuvam Shah's user avatar
0 votes
1 answer
107 views

Oracle in optimization

I have encountered the word oracle in the following context: Given an $\alpha$-approximate oracle for stochastic optimization we show how to implement an $\alpha$-approximate solution for robust ...
Blade's user avatar
  • 129
6 votes
2 answers
7k views

Difference between RMSProp and Momentum?

Can someone please tell me the clear difference between the approaches of RMSProp and Gradient Descent with Momentum ? Both try to achieve the same effect . One of the blogs that I read states the ...
Bharathi's user avatar
  • 277
2 votes
1 answer
566 views

Reversing the input and output of an ML algorithm to Optimize

My dataset consists of multiple input variables (X) and multiple output variables (Y). For example: ...
SineFromAbove's user avatar
4 votes
1 answer
3k views

SGD versus Adam Optimization Clarification

Reading the Adam paper, I need some clarificaiton. It states that SGD optimization updates the parameters with the same learning rate (i.e. it does not change throughout training). They state Adam is ...
Shinobii's user avatar
  • 419
0 votes
0 answers
77 views

Logistic loss increasing while training with minibatches using the adam algorithm

I am trying to write my own code to use the adam algorithm for logistic regression. I am pretty sure It is training correctly as when I run it I am able to accurately classify a bunch of toy data that ...
invader.zimm's user avatar
2 votes
2 answers
1k views

Appropriate loss function for multi-hot output vectors

I have some data in which model inputs and outputs (which are the same size) belong to multiple classes concurrently. A single input or output is a vector of zeros somewhere between one and four ...
duhaime's user avatar
  • 194
1 vote
1 answer
715 views

Google OR-Tools - Routing - Penalties - Adding Different Penalty to Different Location (Python)

I am using Google's OR-Tools for route optimisation. References can be found here. I am performing an optimisation where certain pick-up locations are dropped based on a penalty at each location. The ...
bradS's user avatar
  • 1,615
2 votes
2 answers
518 views

Formal math notation of masked vector

I'm struggling to write my algorithm in a concise and correct way. The following is an explanation for an optimizer's update step of part of a vector of weights (not a matrix in my case). I have a ...
leed's user avatar
  • 145
0 votes
1 answer
43 views

Cat Classifier becomes worse the more you train it

I am using a dataset from kaggle to train a feed forward neural-neteork with no convolutional layers. I wanted to try it this was as a learning exercise with Pytorch without Transfer Learning and ...
Raikan 10's user avatar
0 votes
1 answer
838 views

Hill Climbing Algorithm - Optimum Step Size

I am implementing a standard hill climbing algorithm to optimise hyper-parameters for a predictive model. The hill climbing algorithm is being applied as part of a two-stage approach: Apply grid ...
A_Murphy's user avatar
1 vote
2 answers
826 views

How to tell my neural network that I care much more about precision than recall?

I am training a neural network for a multilabel classification problem, so my last layer consist of n_classes sigmoid neurons. Now, I know that it is impossible to ...
hipoglucido's user avatar
  • 1,170
0 votes
1 answer
29 views

Optimization function returns the same optimal parameters for two labels

I've recently enrolled in the Coursera machine learning, and am working my way through making my own classifier for the Iris dataset problem using matlab. I'm training a classifier for each species (...
Omar El Atyqy's user avatar
0 votes
1 answer
458 views

What is the objective that is optimized with Random Search?

I have recently learned about Random Search (or sklearn.model_selection.RandomizedSearchCV in Python) and was thinking about the theory behind the optimization process. In particular my question is, ...
RazorLazor's user avatar
1 vote
1 answer
785 views

how to compute bernoulli entropy?

I am reading gail implementation code in openai baselines. they compute bernoulli entropy as one of the loss in adversary network loss function. In their code, they implement bernoulli entropy as ...
LucasYang's user avatar
1 vote
0 answers
55 views

Negative impact of "important" features on model performance

I have a random forest regressor with a set of base features, fit & optimised with sklearn random search algorithm. When I add a set of additional features and retrain (again with random search ...
Niggl's user avatar
  • 21
3 votes
0 answers
96 views

Dissecting and understanding the Adam optimization's formula

Adam's optimization has the fololwing parameter update rule : $$ \theta_{t+1} = \theta_{t} - \alpha*\dfrac{m_t}{\sqrt{v_t + \epsilon}}$$ where $$ m_t \text{ is first moment of gradients and} \space ...
black sheep 369's user avatar
3 votes
1 answer
631 views

Intuition behind Adagrad optimization

The following paper ADADELTA: AN ADAPTIVE LEARNING RATE METHOD gives a method called Adagrad where we we have the following update rule : $$ X_{n+1} = X_n -[Lr/\sqrt{\sum_{i=0}^ng_i^2}]*g_n $$ Now I ...
black sheep 369's user avatar
1 vote
0 answers
243 views

Does optimizer highly affect on accuracy?

I used SGD as optimizer and its accuracy result is about 97% and I have changed optimizer to Adam surprisingly, my accuracy became 49% I only changed optimizer and didn`t change anything else but ...
real_noob's user avatar
1 vote
2 answers
137 views

Finding a vector that minimize the MSE of its linear combination

I have been doing a COVID-19 related project. Here is the question: N = vector of daily new infected cases D = vector of daily deaths E[D] = estimation of daily deaths N is a n-dimensional vector, n ...
Jiaming Na's user avatar
1 vote
0 answers
22 views

Optimization problem with different type of constraints

I'm new to optimization problems. I want to find optimum values for my objective function. You can imagine my function as E = f(t1, t2, t3). I want to minimize <...
Babak.Abad's user avatar
1 vote
1 answer
161 views

Best approach to find optimal solution to linear equation by group in R

I am currently modeling a pricing and discount system in R. My data frame looks as follows: ...
Fnguyen's user avatar
  • 1,743
9 votes
2 answers
2k views

Optimising for Brier objective function directly gives worse Brier score than optimising with custom objective - what does it tell me?

I am training an XGBoost model and as I care the most about resulting probabilities, not classification itself I have chosen Brier score as a metric for my model, so that probabilities would be well ...
Xaume's user avatar
  • 192
1 vote
0 answers
12 views

Basket items optimisation minimising constraints

I have a real problem (not home work) when I have to distribute an ordered list by position to respect some constraints eg. 1. 11 2. 15 3. 18 4. 18 5. 1 baskets:...
venergiac's user avatar
  • 111
1 vote
1 answer
475 views

Computing variance of an SGD iteration

It is known that SGD iteration has huge variance. Given the iteration update: $$ w^{k+1} := w^k - \underbrace{\alpha \ g_i(w^k)}_{p^k}, $$ where $w$ are model weights and $g_i(w^k)$ is gradient of ...
user93607's user avatar
1 vote
1 answer
27 views

Is Neural Network Architecture independent of Data?

If I change my dataset (let's say it is always images), should I change the architecture of my neural network?
Angadishop's user avatar
1 vote
1 answer
244 views

Does severe multicollinearity affect solving linear regression by gradient descent?

Since OLS may fail when there is severe/near perfect multicollinearity, how would gradient descent perform in such a scenario? Does it converge at the minima? (My guess is, Cost function of linear ...
Preetham_tsp's user avatar
1 vote
1 answer
96 views

How does the construction of a decision tree differ for different optimization metrics?

I understand how a decision tree is constructed (in the ID3 algorithm) using criterion such as entropy, gini index, and variance reduction. But the formulae for these criteria do not care about ...
sgk's user avatar
  • 111
1 vote
0 answers
17 views

Need ideas on how to find the best email send frequency go get maximum desired action

I have got data which contains email id total times email has been send to him (frequerny) and desired action taken (read, clicked etc), non desired action taken(unsubscribed, etc), also count of no ...
Sourav Roy's user avatar
0 votes
1 answer
11k views

How to Minimize mean square error using Python

I want to minimise mean square error function to find best alpha value (decay rate) for my model. Here is the description of my model: ...
MAC's user avatar
  • 277
1 vote
2 answers
2k views

Differential Evolution optimal tolerance parameter

I am trying to optimize the parameters of a global optimization system for my set of data, because I will have a bunch of similar data to process so I need to fine tune the global optimizator so that ...
user421473's user avatar
5 votes
1 answer
2k views

How many times is backprop used in epoch?

As I understand for the algorithms that use gradient descent we have to pass data to the algorithms multiple times so that the optimum is found. So one epoch means that the forward-backprop (and ...
Alex T's user avatar
  • 153
-3 votes
1 answer
84 views

Solve an equation using machine learning [closed]

Imagine we have the following equation: y=xz. We have y but not other ones. Note that y is like a matrix and we could as many sample we want. It is the values obtained from sensors. This means it ...
Arkan's user avatar
  • 453
1 vote
2 answers
115 views

Reinforcement Learning : Why acting greedily with the optimal value function gives you the optimal policy?

The course of David Silver about Reinforcement Learning explains how you get the optimal policy from the optimal value function. It seems to be very simple, you just have to act greedily, by ...
tristan's user avatar
  • 11
1 vote
1 answer
140 views

ADAM algorithm for multilayer neural network

I’m trying to touch neural networks without using “in box” algorithms. And so I found out that nowhere is written how to calculate square of gradient for hidden layers in ADAM optimizer. I took the ...
Fyodor  Alekhin's user avatar
3 votes
1 answer
65 views

How to optimize client's portafolio with analytical models?

I have a model in which we want to optimize the probability of an outcome depending on a election of some product (a personalized product for every client amongst three posibilities). The product is ...
Juan Esteban de la Calle's user avatar
1 vote
1 answer
116 views

Can we optimize heterogeneous parameters of RBF Network using Gradient Descent?

There're three parameters in the Radial Basis Function Networks (RBFN). Centers of RBFs Width of RBFs Weights of RBFs It's a fact that Weights can be easily updated using a simple Gradient Descent. ...
Tarlan Ahad's user avatar
2 votes
1 answer
72 views

Salesman problem with additional conditions and features [closed]

I'd like to specify the kind of a problem I encountered. I need to compose the best route for a car driver who goes to different cities. His aim to check in the most proper car park in a city taking ...
James Flash's user avatar
3 votes
3 answers
256 views

Constructing function - f(x,y) for the given minimums (Python) [closed]

Problem Statement: I need to construct a function f(x,y) in which there're 3 minimums. 2 local and 1 global which are written below. Locals are: z = f(0.2,0.3) = 0.7 | z = f(0.6,0.8) = 0.8 Global is: ...
Tarlan Ahad's user avatar
0 votes
1 answer
193 views

What applications does linear programming have in data science?

I'm currently learning about linear programming in my degree. I'm wondering how this is relevant to anything in data science?
Data's user avatar
  • 467
5 votes
1 answer
106 views

Can I completely cancel the effects of using a smaller batch size by reducing the learning rate?

I'm having the problem that the data from a regular sized batch (e.g., 32, 64) doesn't fit in my GPU. Among other solutions, I'm considering reducing the batch size, as is normally suggested. Of ...
Nicolas Gervais's user avatar
4 votes
2 answers
5k views

Is it possible to make F1_Score differentiable and use it directly as a Loss function?

One of the metrics that is widely used in binary classification is the F1 score: $F_1 = 2\cdot \frac{recall \cdot precision}{recall+precision}$ The problem of the F1-score is that it is not ...
Carlos Mougan's user avatar
1 vote
0 answers
30 views

How to build Explanatory Graph for Convolutional Neural Network?

I m reading very interesting paper (https://arxiv.org/pdf/1812.07997.pdf) that aims to interpret convolutional neural network using graph. The general idea is when there are co-related parts in layers ...
Hyphen's user avatar
  • 183

1
3 4
5
6 7
10