Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [optimization]

In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).

3
votes
1answer
26 views

Newton method and Vanishing Gradient

I read the article on Vanishing Gradient problem, which states that the problem can be rectified by using ReLu based activation function. Now I am not able to understand that if using ReLu based ...
2
votes
1answer
43 views

Newton's method optimization for Deep Learning

I'm reading this paper "Deep learning via Hessian-free optimization" by J. Martens, I am having difficulty figure out the following statement: In the standard Newton's method, $q_{\theta}(p)$ is ...
1
vote
1answer
59 views

Understanding general approach to updating optimization function parameters

This question not related to a specific method or technique, rather there is a broader concept that I'm struggling to see clearly. Introduction In machine learning, we have loss functions that we're ...
12
votes
4answers
3k views

Is Gradient Descent central to every optimizer?

I want to know whether Gradient descent is the main algorithm used in optimizers like Adam, Adagrad, RMSProp and several other optimizers.
0
votes
1answer
38 views

How should I tackle this real-life hypermarket problem?

I registered myself in the payback program of the hypermarket I am going to. For every 2$ I get 1 point. I buy the same products every week (Feta 2.19\$, Milk 0.99$, ...). I visit only in weekdays. ...
1
vote
1answer
25 views

Optimization based on validation and not training

Hello neural network programmers, I am currently creating a neural network with keras, as I am not that familiar with tensorflow and it's a bit more difficult. I want my optimization to optimize the ...
0
votes
1answer
11 views

What makes a problem good for an evolutionary strategy vs a genetic algorithm vs particle swarm optimization?

I understand that evolutionary strategies (ES), genetic algorithms (GA), and particle swarm optimization (PSO) are all algorithms used to solve optimization types of problems, but what might make an ...
0
votes
1answer
13 views

Can we use decreasing step size to replace mini-batch in SGD?

As far as I know, mini-batch can be used to reduce the variance of the gradient, but I am also considering if we can achieve the same result if we use the decreasing step size and only single sample ...
0
votes
0answers
14 views

How to predict similarity of unseen data to the training set?

I have a time series of human pose data which are recorded from real humans. I want to train the model with unsupervised learning on the training data. Let's call this the "real" training data. The ...
0
votes
1answer
15 views

General question on the approach to optimise numbers

I have a huge huge model in SQL that nobody knows what it is doing. This model spits out some numbers and those numbers should be optimised to match another batch of 'correct' numbers as much as ...
3
votes
1answer
27 views

Can Adagrad be used to optimize non-differentiable functions?

I am reading a book (TensorFlow For Dummies, Matthew Scarpino), and here it says: Adagrad methods compute subgradients instead of gradients. A subgradient is a generalization of a gradient that ...
0
votes
0answers
22 views

How to optimize function built on top of the classifier?

I have a dataset with classification model build for it for $n$ classes as target. And also using the probabilities for each class, which classificator returns, I built confidence function for each ...
6
votes
3answers
82 views

Least Squares optimization

The cost function given as $\hat{\beta} = (Y - \beta X)^T (Y-\beta X)$ is used to evaluate the weights $\beta$. Here $X$ is the data and $Y$ is the output. On taking the derivative, we get the ...
1
vote
1answer
16 views

Grouping numeric data into efficient group/pool

I have devices assigned with different data plan. But based on device behaviour amount of data used by device changes during the month. I need to put the device into appropriate data plan based on ...
2
votes
1answer
54 views

What is the most appropriate machine learning approach for this scenario?

The scenario is pretty simple, and I'm sure it's been done a million times. The problem is i don't know the terminology to find the correct resources on the web. Scenario: I have an environment that ...
2
votes
1answer
24 views

Efficient way to search list of items in a text document

I have a list of items (size ~50K) and several documents( average page per document ~10). I am trying to find what all items are listed in each document as follows : ...
0
votes
0answers
16 views

How to adjust deep learning parameters using Particle swarm optimization (PSO)?

As success of deep learning depends upon appropriately setting of its parameters to achieve high-quality results. The number of hidden layers and the number of neurons in each layer of a deep machine ...
1
vote
0answers
40 views

Can anyone explain how fmincg works?

In the Coursera machine learning course by Andrew Ng, fmincg is used for optimization frequently. I'm not getting the mechanism of it. Can anyone give overview of how it finds the minimum of any ...
3
votes
1answer
52 views

Linear optimization problem of $argmin$

Consider a vector $a \in R^n$. I want to know how I can find analytically the solution of the following optimization problem: $x^* = argmin_{x \in R^n} f(x)$, where $f(x) = ||x-a||_{2}^2 + \lambda ...
0
votes
0answers
14 views

Best way to optimize server runtime for multiple jobs?

I am not well versed in the data science field as I am working on devops now so I wanted to ask this community. If you can refer me to a more proper stackoverflow then let me know! Given servers and ...
1
vote
1answer
15 views

Bayesian optimisation in deeplearning

Has anyone tried using Bayesian optimisation to get best learning rates, and other hyperparameters for deeplearning. How to change the parameters between the training. Any examples on callbacks? Can ...
1
vote
1answer
23 views

Strategies for continuously assessing and improving model performance

I am building a supervised machine learning model to generate forecast. So I would have historic data like this: SKU, Month, .... other features, Actual Volume ...
0
votes
0answers
11 views

Storage optimisation of sparse matrix

I have a question about optimisation of the storage of a matrix. Let's say that I have a 3D matrix representing a cartography of a room. In this matrix, I have values representing for example the ...
0
votes
0answers
55 views

iris dataset feature selection using cuckoo search optimisation

I am trying to use a python library SwarmPackagePy to select significant features from a dataset for example iris dataset But i am stuck in dimension and iteration setting. In addition in how to use ...
1
vote
1answer
24 views

Conjugated gradient method. What is an A-matrix in case of neural networks

I am reading about conjugated gradient methods to understand how they exactly work. I understand that a pair of vector $u$ and $v$ are conjugated with respect to $A$ if $u^TAv=0$. I also read that $A$ ...
0
votes
1answer
33 views

What does it mean when someone says “Most of the data science algorithms are optimization problems”

I was trying to understand the Gradient Descent algorithm from this article and the author says Most of the data science algorithms are optimization problems I come from software engineering ...
1
vote
1answer
41 views

Is it necessary to tune the step size, when using Adam?

The Adam optimizer has four main hyperparameters. For example, looking at the Keras interface, we have: ...
1
vote
1answer
209 views

Recurrent Neural Network (LSTM) not converging during optimization

I am trying to train a RNN with text from wikipedia but I having having trouble getting the RNN to converge. I have tried increasing the batch size but it doesn't seem to be helping. All data is one ...
0
votes
1answer
33 views

error while running lasso.py

The following is the error code generated while running lasso.py. Can anybody help in fixing the same. Here is the code: ...
0
votes
0answers
46 views

Swap memory size increases when running multiple Keras models

I have the following Python code with Keras ...
1
vote
0answers
17 views

What is the simplest optimization algorithm for a multi-parameter closed system?

I'm working on a minimization problem for a wireless communication link. I want to minimize the total bit error rate (BER) of a closed system. My problem is that the signal has multiple (50) ...
1
vote
1answer
27 views

Does policy optimization learn policies to make better actions with higher probability? [closed]

When I talk about policy optimization, it is referred to the following picture, and it is linked to DFO/Evolution plus Policy Gradients. I would like to know is it correct to say: Policy ...
0
votes
1answer
26 views

Minimization algorithm that can consider gradient close to solution

I want to minimize a function which has sharp gradients close to each local minimum. Due to process tolerances, I want to find solutions which meet some minimum criterion (e.g. lower than x), but have ...
0
votes
1answer
193 views

Should the minimum value of a cost (loss) function be equal to zero?

We know optimization techniques search in the space of all the possible parameters for a parameter set that minimizes the cost function of the model. The most well-known loss functions, like MSE or ...
1
vote
1answer
20 views

Are there any learner-specific optimizers?

In reading about machine learning (ML), and working through some basic examples, it appears to me most learning algorithms use generic optimizers. I am using the word "optimizer" to describe the ...
2
votes
1answer
38 views

How to derive the sum-of squares error function formula?

I'm attending a Machine Learning course and I'm studying linear models for classification right now. Slides present approaches to learn linear discriminants (Least squares, Fisher's linear ...
0
votes
1answer
50 views

How to Define a Cost Fucntion?

I want to define a cost function in python to identify optimum value in days when i should end a marketing campaign to save spend on campaigns not generating traffic good traffic. Problem is I dont ...
1
vote
0answers
12 views

How to optimize the separation of two distributions from binary classfication

Given a sample where for each individual a classification is predetermined (e.g. sick or not) and 5 random variables are measured. The random variables are on the same scale but from differnt bins. E....
1
vote
1answer
24 views

AI that maximizes the storage of rectangular parallelepipeds in a bigger parallelepiped

As you can see in the title, I'm trying to program an AI in Java that would help someone optimize his storage. The user has to enter the size of his storage space (a box, a room, a warehouse etc...) ...
0
votes
0answers
44 views

how to calculate number of frames in epsilon greedy with decay rate?

If starting epsilon is alpha and end epsilon is beta in epsilon greedy algorithm. discount rate is gamma and epsilon decay is lambda, how to calculate the F: number of frames to reach from alpha to ...
0
votes
0answers
91 views

What are the reasons of select a optimizer to be SGD or Adam in DQN?Why?

I saw several comparison between SGD, RMSPROP and ADAM but what I am looking for is their comparsion in DQN algorithm? What is best to select as optimizer SGD or Adam in DQN? Why? Please check the ...
0
votes
1answer
170 views

Which are the latest Optimization techniques in artificial intelligence?

My project work is optimization in power system using artificial intelligence (like fault location and classification,load forecasting and context awareness and IoT etc ) and I have used PSO (particle ...
2
votes
0answers
145 views

clipping the reward for adam optimizer in keras

I would like to clip the reward in keras. I saw it is possible to clip the norm and clip the value is sgd as follows: ...
2
votes
1answer
16 views

Algorithm for campaign optimization (Digital Advertising)

Suppose i am running an Ad thru an Ad exchange A, and i have a set of campaigns running on it. I have The spend of the campaign. The budget allocated to it. The number of hours it took to exhaust it'...
1
vote
1answer
28 views

has number of output layer of DNN any effect in speed of find the optimal answer of DNN?

has number of output layer of DNN any effect in speed of find the optimal answer of DNN? For instance the more episodes is needed to train a DNN when the number of outputs is more? Is it correct?
1
vote
0answers
135 views

Policy gradient: why does this converge with Adam and not SGD?

I am looking into policy gradient methods. I stumbled into this implementation: https://gist.github.com/calclavia/cfcd41ad4e47d7b9b6ab8af15410747a It uses a Nesterov Adam optimizer. If I run it, it ...
8
votes
2answers
130 views

Why aren't Genetic Algorithms used for optimizing neural networks?

From my understanding, Genetic Algorithms are powerful tools for multi-objective optimization. Furthermore, training Neural Networks (especially deep ones) is hard and has many issues (non-convex ...
0
votes
1answer
48 views

What is new population in genetic algorithm?

Here is my (mis?)understanding of genetic algorithm: Create n individuals. This is initial population Calculate fitness of each individual in this population ...
3
votes
1answer
62 views

Why imbalanced data-set will bias the prediction model towards the more common class?

As we know, an imbalanced data-set has a disadvantage of training a model for deep learning. However, I don't know how to explain it with mathematics?
2
votes
3answers
126 views

Small dataset in Time series

I have soccer data with a time series index. 30 seconds interval. So, 194 rows for 90+ minutes per game. I have 1500 games. The dataframe has the following information. Home/Away: • Goal Total. • ...