Questions tagged [optimization]

In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

Filter by
Sorted by
Tagged with
1
vote
0answers
22 views

XGBoost speed issues

I'm trying to optimize the hyperparameters for XGBoost, thus needing to run it multiple times with different parameters. However the time needed to run single XGBoost with the parameters provided ...
0
votes
1answer
34 views

Non-Convex Constraints for Classification Problems

I am willing to create a hypothetical non-convex constraints for the purpose of practising nonlinear classification using an algorithm. I thought of such constraints in the form: $x^TAx + Bx \leq c$. ...
1
vote
1answer
19 views

How can we conclude that an optimization algorithm is better than another one

When we test a new optimization algorithm, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,i.e., in terms of accuracy, f1 ...
0
votes
0answers
14 views

How to implement AdamW?

I have implemented AdamW but I am not getting good results, is there some mistake in my implementation? ...
2
votes
0answers
22 views

Optimizer for Function Approximation using Fully connected Neural Network

In short, my query is: Which optimizer(s) should one choose to experiment for a fully connected neural network, if she wants perfect fitting (mae < 1e-04) on the training data? Details: In my ...
1
vote
1answer
29 views

Gaussian Mixture Models Clustering

When using the EM algorithm in Gaussian Mixture Models (GMM), in the E-step, we take each x set in the training dataset to calculate and update the "weight" and parameters of each Gaussian ...
0
votes
0answers
10 views

Custom layer in Keras and optimizer

How is optimizer related to our own Keras layer? Do we have to rewrite optimizer for that certain layer? For example, suppose we have ...
0
votes
0answers
29 views

Optimizing parameters for an image-generation algorithm

I have a program that takes an image and a list of parameters and generates a new image. I would like to automate the selection of parameters to produce the 'best' image. 'Best' in this context doesn'...
1
vote
3answers
57 views

Gradient descent in a noisy environment

How to know the right direction in a noisy environment? In the typical example of neural network learning, we can see several local minima. The gradient descent is choosing one local minimum and ...
0
votes
1answer
12 views

Python metaheuristic packages

I need to use a metaheuristic algorithm to solve an optimization problem on a Python codebase. Metaheuristics usually need to be written in C++ or Java as they involve a lot of iterations, while ...
0
votes
0answers
32 views

Why does degradation occur in deep neural networks?

It has been shown that "plain" neural networks tend to have an increased amount training error, and accompanied test error, as more layers are added. I am not quite certain as to why this occurs. In ...
0
votes
0answers
16 views

Algorithm/Model for Power System Optimisation

For my engineering honours project, I'm performing a study of voltage control on the electrical distribution network, using reactive power provision from residential solar inverters. Basically, each ...
0
votes
0answers
9 views

Are there any other recommended optimizers for word2vec/glove than Adagrad and SparseAdam?

adagrad and sparseAdam work great for sparse training because there’s separate sums for each of the parameters. Are there any other recommended optimizers for embedding training?
0
votes
1answer
19 views

Can we Create Neural network(Simple one such as Multi Layer perceptron) that only contains positive weights only?

I was wondering if there is a specific method to create a well performing neural network with only positive weights (I already tried clipping the weight before training or so and initializing the ...
0
votes
1answer
39 views

How to deal with a constant value as an output from neural network?

I am using feedforward neural network for regression and what I get as a result of prediction is a constant value visible on the graph below: Data I use are typical standardised tabular numbers. The ...
1
vote
2answers
37 views

Different optimizers for generator and discriminator in GAN

I've seen an advice about GAN implementation, that there should be different optimizers for generator (G) and discriminator (D). As I understand, it depends on how fast each model (G and D) ...
1
vote
1answer
48 views

Suggestions for Matchmaking Algorithm

I run a heterosexual matching making service. I have my male clients and my female clients. I need to pair each of my clients with their "soul mate" based on several attributes (age, interests, ...
1
vote
2answers
26 views

Equation related to Smoothness

If you have a differentiable function $f:\mathbb{R}^d\rightarrow\mathbb{R}$ that is $\beta$-smooth (for all $v$ and all $w$, you have $\|\nabla f(v)-\nabla f(w)\| \leq \beta \|v-w\|$), how can you ...
0
votes
0answers
7 views

Backtracking Line search for Multiclass classification gradient descent

For my case i am dealing with multiclass problem and there are total 28 direction component for each class and there are total 5 classes, for given equation above, f(w+nd) and f(w) gives scaler values ...
0
votes
0answers
9 views

How to handle optimization problem when objective function and constraints involve different set of parameters

I am working on this constrained optimization problem. The objective function is the efficiency of the machine which is determined by 6 controllable variables. The constraint is the pressure can't ...
2
votes
1answer
25 views

Quadratic approximation of L1 regularized cost function

I'm reading the Deep Learning book of Goodfellow, but I fail to see why minimization of (7.22) gives (7.23). I tried to compute the gradient w.r.t. the $w_{i}$ and set this to zero, but it doesn't ...
1
vote
1answer
23 views

Is hyperparameter tuning more affected by the input data, or by the task?

I'm working on optimizing the hyperparameters for several ML models (FFN, CNN, LSTM, BiLSTM, CNN-LSTM) at the moment, and running this alongside another experiment examining which word embeddings are ...
0
votes
0answers
12 views

Distributed optimization with a common variable among individual agents

I am new to optimization techniques and have a doubt in my approach. Consider I have an agent which tries to optimize on 3 variables C1,x11,x12 to minimize power. I have 60 such agents which ...
0
votes
1answer
25 views

What is the best way to optimize the parameters in a Sklearn classifier, when I have little data?

What is the best way to optimize the parameters in a Sklearn classifier when I only have a data set with 684 rows and 177 columns, and the column I want to predict has 3 labels? I know I should split ...
1
vote
1answer
20 views

Methodology for parallelising linked data?

If I have some form of data that can have inherent links to all other data in the set but I wish to parallelise out this data in order to increase computation time or to reduce the size of any ...
0
votes
2answers
108 views

How to use Chi-square test in dataset with negative values

I could not fully explain the title. In order to use the Chi-square test in my dataset, I am finding the smallest value and add each cell with that value. (for example, the range of data here is [-8,...
0
votes
2answers
78 views

Python lifelines - ConvergenceWarning: Newton-Raphson failed to converge sufficiently in Cox prop hazard

When calling CoxPHFitter() on my full dataset I'm getting the following error: ...
0
votes
0answers
4 views

What are the main reasons which does not cause the training error of yolov2 to not diminish?

I am using https://github.com/thtrieu/darkflow yolov2 for detecting and classifying the images of passport. There are 8 classes and all the objects are passport details like name, father name, mother ...
6
votes
2answers
86 views

Why is taking the gradient of the average error in SGD not correct, but rather the average of the gradients of single errors?

I am a little confused about taking averages in cost functions and SGD. So far I always thought in SGD you would compute the average error for a batch and then backpropagate it. But then I was told in ...
0
votes
2answers
17 views

Why does discriminiator accuracy falls to 0%, and is there a fix around this?

I am training a Vanilla-GAN(or original GAN 2016) on a pokemon dataset https://www.kaggle.com/kvpratama/pokemon-images-dataset, for few epochs the discriminator has 100% accuracy over the real ...
1
vote
1answer
142 views

scipy.optimize.minimize(method=’trust-constr’) doesn't terminate on xtol condition

I have set up an optimization problem with linear equality constraints as follows ...
0
votes
0answers
11 views

Resources on Non-Linear Programming and Optimization

With respect to the below question. https://math.stackexchange.com/questions/871370/optimizing-independent-variables-to-maximize-dependent-variable/871421#871421 I am currently building a similar ...
0
votes
0answers
11 views

How to optimize MAPE in regression algorithms

I have a regression task where the label is varying from about 0.001 to 1000. One of the feature called group, for example, group A corresponding label from 0-0.1 and group G corresponding label from ...
4
votes
1answer
214 views

Mathematical formulation of Support Vector Machines?

I'm trying to learn maths behind SVM (hard margin) but due to different forms of mathematical formulations I'm bit confused. Assume we have two sets of points $\text{(i.e. positives, negatives)}$ one ...
1
vote
1answer
30 views

RMSProp - To a layman?

Can someone explain RMSprop in layman's terms? I have tried reading various resources but they don't talk about the intuition on couple of things: 1) Why is is different from SGD with momentum? 2) ...
1
vote
0answers
9 views

Why does NAG cause unstable validation loss?

I'm building a neural network for a classification problem. When playing around with some hyperparameters, I was surprised to see that using Nesterov's Accelerated Gradient instead of vanilla SGD ...
0
votes
0answers
4 views

How calculate computation time for each part of the network

I want to report how much times it takes to compute each specific part of the network in a batch (forward and backward time). For example, in this paper they've reported RNN, softmax, and optimization ...
1
vote
2answers
31 views

Parallel hyperparameter optimization techniques?

Most hyperparameter optimization technique want to evaluate points one by one. I have an expensive optimization problem, but i can run hundreds of evaluations in parallel. The dimension of the problem ...
0
votes
0answers
14 views

Find possible fearure values to predict a certain outcome

I have a dataset about patients waiting times in a health district. The data is aggregated by health provider, type of medical procedure, urgency of the procedure (3 classes) and reports the n. of ...
1
vote
3answers
57 views

Different learning rates for each dimension

I have been thinking about why normalization and scaling are done for each feature in the basic context of gradient descent. One thing that got me wondering is that we use a pre-defined set of ...
0
votes
0answers
32 views

Optimizing Market Basket Analysis by limiting threshold

I'm creating a suggestion model through MBA. I observed that in my particular model, that if the min_support was placed as 0, the model would take an insanely long ...
1
vote
2answers
63 views

How to find the right regression model for my project

I tried several things to find the best regression model with the best parameters but i can't go higher than 40% right predictions. So i have 67741 rows in an excel file. the data looks like this ...
1
vote
1answer
60 views

What should I study to find optimal value of best feature combinations in machine learning?

I would like to do production optimization with machine learning and/or optimization problem. My goal is not to find minimizing loss in loss function only to give the best y value. My ultimate goal ...
0
votes
0answers
16 views

Techniques to increase stability of parameters

I have a highly non-linear function $f(x,y,t)$, which I am trying to maximise over different time periods $t$. It is the case that other parameters are of mixed type with $x \in Re$ and $y \in {Z}$, ...
1
vote
0answers
12 views

How can I evaluate the constant parameters of a series of linear functions with varying input parameters?

I have the following setup of linear functions: g0(x0, y0) -> (x1,y1) g1(x1, y1) -> (x2, y2) ... about 30 more cases Where gi is a function parameterized by 6 values (ai-fi). Which can be described ...
0
votes
0answers
26 views

How to optimise a boolean expression

I am working on an optimization problem involving Boolean expressions and wanted some help as I have very little knowledge about the topic. The problem statement is as follows: There are a set of ...
1
vote
1answer
27 views

How Stochastic Gradient Descent used like Mini Batch gradient descent?

As I know Gradient Descent has three variants which are: 1- Batch Gradient Descent: processes all the training examples for each iteration of gradient descent. 2- Stochastic Gradient Descent: ...
0
votes
1answer
20 views

How to find optimized x values (input features) after training in deep learning?

I did deep learning training by Keras. I have done the training part by model.fit If I do model.predict, it only gives me y value. But I want to know x (input features) that gives the best y value, ...
1
vote
2answers
33 views

Is there any advantage in using Particle Swarm Optimization for clustering than K-Means?

I have read some paper about using particle swarm optimization. It doesn't look give much different result than K-Means. I tried to use PSO for clustering but the result is pretty much the same with K-...
0
votes
1answer
72 views

Why are parameter updates downscaled by uncentered variance (instead of centered variance) in Adam optimizer?

In Adam optimizer algorithm, parameter updates are computed as follows: $\theta_t \leftarrow \theta_{t-1} - \alpha \frac{\hat{m}_t}{\sqrt{\hat{v}_t}+\epsilon}$ Where $\hat{m}_t$ is a bias-corrected ...