Questions tagged [optimization]

In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

Filter by
Sorted by
Tagged with
1
vote
2answers
31 views

“Super” Optimizer concept

I was wondering why there isn't a feature built into common-use ML libraries, like Keras, that plugs many different combinations of layers and nodes to multiple models and trains them simultaneously ...
0
votes
0answers
14 views

Assumptions on discounted long-term loss

The infinite horizon discounted long-term loss is defined as: $$ f(\theta) = \mathbb{E}_{\tau \sim \mathbb{P}(.|\theta)}\left[\sum_{t=1}^{\infty}{\gamma^t l_m(s_t,a_t)}\right]$$ where $(s_t,a_t) \in ...
1
vote
0answers
8 views

optimal minimization algorithm for platou

Greeting, I'm trying to solve an optimization problem (minimization, to be specific). My problem is that my function has one major plateau (see example image). I'm using the optimization algorithms ...
1
vote
0answers
20 views

Algorithms and Techniques used for Route Optimization

What are the algorithms/techniques used for route optimization problems like VRP in Data Science? Vehicle Routing Problem (VRP) can be described as the problem of creating a set of optimal routes ...
0
votes
0answers
10 views

Is using cross-entropy enough to ensure the output is a distribution probability?

I am following along https://pytorch.org/tutorials/beginner/finetuning_torchvision_models_tutorial.html. In this code, the last layers of the pretrained networks are linear. The loss used in this ...
0
votes
0answers
9 views

Broadcast error in optimize.minimize

I've defined a function that references an array and broadcasts variables across that array. When the function is run, it works fine. However, when I attempt to use scipy.minimize to minimize the ...
0
votes
1answer
15 views

Calculating possible number of configuration

I am wondering how did they get the $19200$ possible configurations? Like, $5^6 = 15625$, where $6$ is the number of hyper-parameters:
1
vote
0answers
24 views

Unsupervised Function Optimization using Input and Output for Loss Function?

I have some vectors {$\mathbf{X_1 ... X_n}$} and they are all of dimension 1 x N. Vectors {$\mathbf{X_1' ... X_n'}$} are also 1 x N and are related to {$\mathbf{X_1 ... X_n}$}, but the relation cannot ...
0
votes
1answer
19 views

which scoring function for validation_curve (regression)?

Is there any thumb of rule which scoring function should be used for e.g. the validation_curve? Atm I try to study the difference between several optimizers: ...
0
votes
0answers
24 views

Variability in CNN test results

I'm trying to do some time series analysis on 1-minute forex data using a CNN. I'm new to deep learning and just getting started in building a model. So this is probably a very basic question, but I'...
1
vote
2answers
23 views

Optimization of pandas row iteration and summation

i'm wondering if anyone can provide some input on improving the speed and calculations of a pandas result. What i am trying to obtain is a summation of IDs in one table (player table) based on each ...
0
votes
2answers
70 views

what is the difference between euclidean distance and RMSE?

I'm searching for a loss function that fits my Project. Actually I have two question but they are in the same direction. I take a look at the definition of the root mean squared error and the ...
1
vote
0answers
8 views

How to explain local minima found between two trained Neural networks?

I have trained 2 neural networks with SGD and then I have taken a linear path between their weights. Say W_0 and W_1 are the weight matrices of network 1 and network 2, respectively. Then I compute ...
1
vote
2answers
26 views

How to optimize input parameters given target and scoring parameters

I'm new to machine learning/optimization, so I apologize in advance if this has been answered before. I don't know which search terms to use. I have a large dataset where I have a number of input ...
0
votes
0answers
10 views

Gym Cartpole not solving with Cross Entropy Method?

Cross Entropy Method is considered as one of the simplest optimization algorithm which can be used for training an agent. I tried to train an agent to solve gym's cartpole environment and I have used ...
3
votes
1answer
78 views

Grid search or gradient descent?

Assume we have a neural network and one if its activation functions is a function of parameter a. We want to find the weights and parameter a that leads to the minimum loss on the validation set which ...
5
votes
3answers
506 views

Genetic algorithms(GAs): to be considered only as optimization algorithms? Are GAs used in machine learning any way?

As a quick question, what are genetic algorithms meant to be used for? I read somewhere else that they should be used as optimization algorithms (similar to the way we use gradient descent to optimize ...
0
votes
0answers
5 views

Is there any relation between binary/ternary quantization using in deep learning and fuzzy?

I am new with binary/ternary quantization but its structure seems to have some relation with fuzzy. Am I in right way? Is there any relation between binary/ternary quantization using in deep learning ...
1
vote
0answers
11 views

If $\ell_0$ regularization can be done via the proximal operator, why are people still using LASSO?

I have just learned that a general framework in constrained optimization is called "proximal gradient optimization". It is interesting that the $\ell_0$ "norm" is also associated with a proximal ...
1
vote
0answers
26 views

XGBoost speed issues

I'm trying to optimize the hyperparameters for XGBoost, thus needing to run it multiple times with different parameters. However the time needed to run single XGBoost with the parameters provided ...
0
votes
1answer
39 views

Non-Convex Constraints for Classification Problems

I am willing to create a hypothetical non-convex constraints for the purpose of practising nonlinear classification using an algorithm. I thought of such constraints in the form: $x^TAx + Bx \leq c$. ...
1
vote
1answer
24 views

How can we conclude that an optimization algorithm is better than another one

When we test a new optimization algorithm, what the process that we need to do?For example, do we need to run the algorithm several times, and pick a best performance,i.e., in terms of accuracy, f1 ...
0
votes
0answers
15 views

How to implement AdamW?

I have implemented AdamW but I am not getting good results, is there some mistake in my implementation? ...
2
votes
1answer
32 views

Optimizer for Function Approximation using Fully connected Neural Network

In short, my query is: Which optimizer(s) should one choose to experiment for a fully connected neural network if she wants perfect fitting (mae < 1e-04) on the training data? Details: In my ...
1
vote
1answer
33 views

Gaussian Mixture Models Clustering

When using the EM algorithm in Gaussian Mixture Models (GMM), in the E-step, we take each x set in the training dataset to calculate and update the "weight" and parameters of each Gaussian ...
0
votes
0answers
22 views

Custom layer in Keras and optimizer

How is optimizer related to our own Keras layer? Do we have to rewrite optimizer for that certain layer? For example, suppose we have ...
0
votes
0answers
29 views

Optimizing parameters for an image-generation algorithm

I have a program that takes an image and a list of parameters and generates a new image. I would like to automate the selection of parameters to produce the 'best' image. 'Best' in this context doesn'...
1
vote
3answers
71 views

Gradient descent in a noisy environment

How to know the right direction in a noisy environment? In the typical example of neural network learning, we can see several local minima. The gradient descent is choosing one local minimum and ...
0
votes
1answer
17 views

Python metaheuristic packages

I need to use a metaheuristic algorithm to solve an optimization problem on a Python codebase. Metaheuristics usually need to be written in C++ or Java as they involve a lot of iterations, while ...
0
votes
0answers
66 views

Why does degradation occur in deep neural networks?

It has been shown that "plain" neural networks tend to have an increased amount training error, and accompanied test error, as more layers are added. I am not quite certain as to why this occurs. In ...
0
votes
0answers
17 views

Algorithm/Model for Power System Optimisation

For my engineering honours project, I'm performing a study of voltage control on the electrical distribution network, using reactive power provision from residential solar inverters. Basically, each ...
0
votes
0answers
12 views

Are there any other recommended optimizers for word2vec/glove than Adagrad and SparseAdam?

adagrad and sparseAdam work great for sparse training because there’s separate sums for each of the parameters. Are there any other recommended optimizers for embedding training?
0
votes
1answer
19 views

Can we Create Neural network(Simple one such as Multi Layer perceptron) that only contains positive weights only?

I was wondering if there is a specific method to create a well performing neural network with only positive weights (I already tried clipping the weight before training or so and initializing the ...
0
votes
1answer
39 views

How to deal with a constant value as an output from neural network?

I am using feedforward neural network for regression and what I get as a result of prediction is a constant value visible on the graph below: Data I use are typical standardised tabular numbers. The ...
1
vote
2answers
70 views

Different optimizers for generator and discriminator in GAN

I've seen an advice about GAN implementation, that there should be different optimizers for generator (G) and discriminator (D). As I understand, it depends on how fast each model (G and D) ...
1
vote
1answer
104 views

Suggestions for Matchmaking Algorithm

I run a heterosexual matching making service. I have my male clients and my female clients. I need to pair each of my clients with their "soul mate" based on several attributes (age, interests, ...
1
vote
2answers
28 views

Equation related to Smoothness

If you have a differentiable function $f:\mathbb{R}^d\rightarrow\mathbb{R}$ that is $\beta$-smooth (for all $v$ and all $w$, you have $\|\nabla f(v)-\nabla f(w)\| \leq \beta \|v-w\|$), how can you ...
0
votes
0answers
7 views

Backtracking Line search for Multiclass classification gradient descent

For my case i am dealing with multiclass problem and there are total 28 direction component for each class and there are total 5 classes, for given equation above, f(w+nd) and f(w) gives scaler values ...
0
votes
0answers
10 views

How to handle optimization problem when objective function and constraints involve different set of parameters

I am working on this constrained optimization problem. The objective function is the efficiency of the machine which is determined by 6 controllable variables. The constraint is the pressure can't ...
2
votes
1answer
35 views

Quadratic approximation of L1 regularized cost function

I'm reading the Deep Learning book of Goodfellow, but I fail to see why minimization of (7.22) gives (7.23). I tried to compute the gradient w.r.t. the $w_{i}$ and set this to zero, but it doesn't ...
1
vote
1answer
23 views

Is hyperparameter tuning more affected by the input data, or by the task?

I'm working on optimizing the hyperparameters for several ML models (FFN, CNN, LSTM, BiLSTM, CNN-LSTM) at the moment, and running this alongside another experiment examining which word embeddings are ...
0
votes
0answers
13 views

Distributed optimization with a common variable among individual agents

I am new to optimization techniques and have a doubt in my approach. Consider I have an agent which tries to optimize on 3 variables C1,x11,x12 to minimize power. I have 60 such agents which ...
0
votes
1answer
29 views

What is the best way to optimize the parameters in a Sklearn classifier, when I have little data?

What is the best way to optimize the parameters in a Sklearn classifier when I only have a data set with 684 rows and 177 columns, and the column I want to predict has 3 labels? I know I should split ...
1
vote
1answer
20 views

Methodology for parallelising linked data?

If I have some form of data that can have inherent links to all other data in the set but I wish to parallelise out this data in order to increase computation time or to reduce the size of any ...
0
votes
2answers
128 views

How to use Chi-square test in dataset with negative values

I could not fully explain the title. In order to use the Chi-square test in my dataset, I am finding the smallest value and add each cell with that value. (for example, the range of data here is [-8,...
0
votes
2answers
142 views

Python lifelines - ConvergenceWarning: Newton-Raphson failed to converge sufficiently in Cox prop hazard

When calling CoxPHFitter() on my full dataset I'm getting the following error: ...
0
votes
0answers
12 views

What are the main reasons which does not cause the training error of yolov2 to not diminish?

I am using https://github.com/thtrieu/darkflow yolov2 for detecting and classifying the images of passport. There are 8 classes and all the objects are passport details like name, father name, mother ...
6
votes
2answers
136 views

Why is taking the gradient of the average error in SGD not correct, but rather the average of the gradients of single errors?

I am a little confused about taking averages in cost functions and SGD. So far I always thought in SGD you would compute the average error for a batch and then backpropagate it. But then I was told in ...
0
votes
2answers
33 views

Why does discriminiator accuracy falls to 0%, and is there a fix around this?

I am training a Vanilla-GAN(or original GAN 2016) on a pokemon dataset https://www.kaggle.com/kvpratama/pokemon-images-dataset, for few epochs the discriminator has 100% accuracy over the real ...
1
vote
1answer
274 views

scipy.optimize.minimize(method=’trust-constr’) doesn't terminate on xtol condition

I have set up an optimization problem with linear equality constraints as follows ...