Questions tagged [optimization]
In statistics this refers to selecting an estimator of a parameter by maximizing or minimizing some function of the data. One very common example is choosing an estimator which maximizes the joint density (or mass function) of the observed data referred to as Maximum Likelihood Estimation (MLE).
496
questions
0
votes
0
answers
27
views
Setting derivative to 0 when minimizing expected loss over possible data samples
Suppose, given $x\in \mathbb{R}^d$, you want $\theta$, which is the solution of
$$
\text{argmin}_{\theta} \mathbb{E}_a[L(\theta;x,a)]
$$
where $a \sim \mathcal{N}(x,\Sigma)$ and $L$ measures a (convex)...
1
vote
1
answer
24
views
Which machine learning models are rational to use on NP-hard and NP-complete "theoretical" problems?
Time and time again I run into "surprising" NP-hard problems that seem naturally simpler than they are. I recently worked on a weighted graph theoretical problem where the point is to ...
- 13
5
votes
1
answer
311
views
What do we mean by optimizer.zero_grad()
This should be a simple question. But it is vague to me. What do we mean by optimizer.zero_grad(). Consider SGD as an example: $W^{t+1}= W^{t}- \lambda g_t$. Which one becomes zero for each batch. It ...
- 57
0
votes
0
answers
22
views
Is there anyway to evaluate the estimation results of least square
Consider the scenario where a practical problem is tackled utilizing the method of least squares. Upon each iteration, an estimation of the parameter $\theta$ is derived via $\hat{\theta} = (X^\top X)^...
0
votes
0
answers
20
views
Feeding more data to a neural network
I watched a video on Tesla's FSD where the drive was really smooth but required one intervention when the traffic light changed to green but the car wouldn't go because it looked like the light was ...
- 1
0
votes
1
answer
59
views
Optimizing decision tree
I have a question regarding the technique/technology which could be applied for the issue:
Suppose I have a rule-based tree or decision tree which predicts a variable Y based on variables A,B,C. This ...
- 1
0
votes
1
answer
26
views
Optimization problem of find a pair of values (x, y) such that they produce an output z
Problem
I am trying to create a process that determines the rent and purchase price of a home that gives me a capitalization rate of 0.08 for instance.
Background
Normally I calculate the ...
- 11
0
votes
0
answers
34
views
How to work with multiple feature types on autoencoder?
This is my first post here.
I am working on an adversarial autoencoder that receives different features, encodes them, and decodes them.
For instance, suppose you have a dataset from a large survey ...
1
vote
2
answers
60
views
weighting voting classifier (MAE and MSE)
I am trying to optimize the weights of a Voting Regressor problem. To achieve the best score, I am considering both MAE and MSE as parameters, using the following formula:
score = w * MAE + (w-1) * ...
0
votes
0
answers
8
views
Team building algorithm based on rank order preferences
I have a dataset with employees at four levels of training: junior employee, senior employee, junior manager, senior manager. I am looking to match them into teams of 4, each team with 1 person from ...
0
votes
0
answers
8
views
Are Julia ML libraries inherently parallelized and optimized?
I am trying to decide if I should learn Julia or stick with Python and its libraries, e.g. TensorFlow and FastAI. One important consideration is whether Julia has a library with similar capabilities ...
- 113
1
vote
2
answers
101
views
Gradient Descent: Is the magnitude in Gradient Vectors arbitrary?
I am only just getting familiar with gradient descent through learning logistic regression. I understand the directional component in the gradient vectors is correct information derived from the slope ...
- 11
0
votes
0
answers
32
views
How does non-negative constrained optimization work?
I am dealing with a machine learning problem in which a logistic regression model is trained with under-bound constrained optimization for model interpretability purposes.
In a simple 2-dimensional ...
0
votes
0
answers
52
views
PySpark Logistic regression model weights are inconsistent between runs
I am training a pyspark logistic regression model using pyspark mllib. I am noticing that the weights are not being consistent in between runs. I have set the random seed in the training script and ...
0
votes
0
answers
6
views
How to interpet the bolded lines in bayesopt
I am using bayesopt to maximize a function that is everywhere less or equal to zero $(f(x) \leq 0)$. The score is essentially a negated Mean Absolute Error because the default behavior of bayesopt is ...
- 93
0
votes
0
answers
12
views
Question about Keras implementation of ADAM
I have a question regarding this line in the Keras implementation of Adam:
alpha = lr * tf.sqrt(1 - beta_2_power) / (1 - beta_1_power)
From the algorithm here, is this step doing the bias correction?...
- 211
2
votes
0
answers
15
views
Consistency error in visualization of policy improvement in Sutton & Barto's book?
Sutton & Barto introduce in their foundational book on "Reinforcement Learning: An Introduction" in the context of Dynamic Programming algorithms for policy evaluation and improvement. ...
- 41
1
vote
2
answers
31
views
Simplex method for equality optimization
I have a linear model that goes as 0.1*x1 + 0.8*x2 + 3.4*x3 + 5.0*x4 + c and this linear model was generated by using a Linear Regression.
MAE is ~ 0.4
MSE is ~ 0....
- 111
1
vote
1
answer
35
views
Is it common that a model only learns in a few training runs and doesn't learn in the majority of training runs
I have a model that does binary classification. I train the same model many times, say, 20 times. In these 20 training runs:
roughly 10% of the training attempts learn fast and the model performs ...
- 153
0
votes
0
answers
47
views
Equivalence between K-Means objective functions proof
I am working on a project involving the K-Means clustering algorithm, and I am trying to prove the equivalence between different formulations of the objective function. Specifically, I want to show ...
0
votes
0
answers
34
views
0
votes
0
answers
24
views
Longer DNN training times when using evolutionary algorithms
I am comparing my deep neural network (DNN) performance when using 2 types of optimizers: gradient-based Adam (properly tuned) and a population-based optimization algorithm (e.g., genetic algorithm (...
0
votes
0
answers
44
views
Multiprocessing data loading in colab
I want to convert mozilla common voice dataset from mp3 to wav. But this dataset is large and convertion takes many time. How can I make this convertion in colab with multiprocessing to decrease time ...
0
votes
0
answers
14
views
How can I use a predicted machine algorithm model to find specific inputs?
I am relatively new to machine learning and now I am working on my thesis regarding that!
My goal is to find a prediction model to see if the strain of the three zones is similar to each other or not.
...
- 1
2
votes
1
answer
26
views
What can we do in order to find customers that follow a specific pre-defined pattern?
Let's say I have a few customers buying product A. How can I find other customers that have the same characteristics, or something close to, these A buyers?
I was thinking about using clustering ...
1
vote
1
answer
15
views
What are the constants in this formula for polytrophic head?
Polytrophic head can be expressed as H = b1N^2 + b2NQ + b3Q^2 where b1, b2 and b3 are constants, N is the speed of the compressor (rmp), and Q is the volumetric flow rate of natural gas at the ...
- 175
0
votes
1
answer
25
views
AdaGrad denominator exponents comparison
Adagrad adapts the learning rate $ \alpha $ all along the gradient descent process, by dividing each weight on a quantity based on the sum of the previous squared gradient up to time $ t $. Therefore, ...
0
votes
1
answer
35
views
Statistical test for comparing number of clusters in data
I am performing $K$-means clustering on a dataset consisting of $n$ observations and $d$ variables, and I'm trying to determine the optimal number of clusters. Is there a test that can determine the ...
- 121
0
votes
0
answers
13
views
how to optimise discount from revenue taking into account number of sales for each discount
I've been working on a problem which I can't seem to get my head round at the moment. I have a dataset lets say is like the following:
user_id
discount_offered
price_pre_discount
price_after_discount
...
1
vote
1
answer
67
views
Which of 2 options is better practice for model optimization: 1) Nested CV wrongly averaging inner CV scores. 2) Two successive CVs on X_all. Altrntv?
Goal: Compare preprocessing methods, models, and hyperparameters without leaking into the final generalization estimate, applying cross-validation (cv), i.e. NOT applying any fixed train/test splits.
...
- 11
1
vote
1
answer
126
views
Solve optimization problem with machine learning algorithm
I have an optimization problem that I solved with grid search using hyperopt in python. In this problem, I have some parameters and a score. I want to find the best ...
0
votes
0
answers
14
views
Why are decision trees driven by the Gini impurity as opposed to the accuracy? [duplicate]
It seems that most implementations of decision trees use the Gini impurity as their partitioning criterion. Why isn't accuracy used instead, since it's a more widespread metric across different ...
- 203
0
votes
0
answers
537
views
How to improve the speed and accuracy of spacy-based named entity recognition (NER)?
I have a file with a couple of million sentences, all in lower-case (I cannot access the cased version). The problem is that the dataset contains a lot of human names and I would like to replace those ...
1
vote
0
answers
86
views
How can I make my neural network learn faster?
I would like to train an LSTM-based variational autoencoder on a large dataset (37 million sentences). However, I have calculated that my training speed as of now is too slow (on Google Colab). I am ...
0
votes
1
answer
197
views
Understanding the stochastic average gradient (SAG) algorithm used in sklearn
For pedagogical purposes I've been trying to create my own implementation of the stochastic average gradient (SAG) algorithm in a logistic regression framework. Page 10 of the associated paper ...
0
votes
1
answer
31
views
What is the best way to use three different losses on two classifiers?
Two classifiers need to be trained simultaneously, and I have three losses, as shown in the figure. Classifiers 1 and 2 will be updated by losses 1 and 2. Furthermore, loss 3 should update the two ...
0
votes
0
answers
19
views
How to decide a State for Deep Q Learning for Production Line scheduling
There is a production floor with W workstations and N jobs with M operations( different processing times per operation ). A job is completed only if its M Operations are completed. Objective is to ...
- 1
1
vote
0
answers
197
views
How Adam optimizer influence the learning rate? [closed]
I read some papers about how ADAM optimizer works, and there are some issues which seems that are confusing:
ADAM equations are:
...
- 363
0
votes
0
answers
7
views
Solving constraint optimization with alternate gradient descent optimization
Problem Setup: Offline contextual bandit with logs of form - <input_context, action, reward>.
Model: A logging/behavior policy $\pi_0$ is used to collect the log data, with context $x_i$, action ...
- 3,775
0
votes
0
answers
46
views
SGD for Graph Neural Networks
I was going through some research papers about Graph Neural Networks; what struck me is that very often SGD is used as optimiser (as in PointGNN, DGCNN and Graphsage).
I figured that for "regular&...
- 11
1
vote
1
answer
507
views
Does settings $\beta_1 = 0$ or $\beta_2 = 0$ means that ADAM behaves as RMSprop or Momentum?
I read on ADAM optimizer, and I saw multiple quotes which say that ADAM is a combination of Momentum and RMSprop optimizers.
So if we:
Set $\beta_1 = 0$ does it means that ADAM behaves exactly as ...
- 363
0
votes
1
answer
25
views
Are there any error functions with imbalanced negative/positive impact
I have a regression task, where positive error should be much worse than negative one. It means the importance of positive error bigger. For example, If real value is less than predicted one weights ...
- 23
0
votes
1
answer
101
views
Why do neural networks with more layers perform better?
Why do neural networks with more layers perform better than a single layer MLP with a number of neurons that leads to the same number of parameters?
I read this post:
https://www.quora.com/Why-do-...
- 363
0
votes
1
answer
374
views
Why does Adam outperform SGD in logistic regression?
I am training a logistic regression model. In case it matters, the features are 1376-dimensional embeddings output from a neural network. I tried both SGD and Adam with a learning rate of $10^{-3}$ ...
- 113
2
votes
0
answers
44
views
Can I find the input that maximises the output of a Neural Network?
So I trained a 2 layer Neural Network for a regression problem that takes $D$ features $(x_1,...,x_D)$ and outputs a real value $y$. With the model already trained (weights optimised, fixed), can I ...
0
votes
1
answer
40
views
Quadratic approximation, second-order optimization method, Newton method
I am learning Newton's method for second-order optimization in ML. I encountered this formula, but I do not understand how we get it. I guess it is from the Taylor series, but I still cannot fully ...
0
votes
0
answers
43
views
Optimization on a convex function used in a loss function
I am currently creating a deep learning model which deals with classification and regression problem together such that each class has continuous value within an interval of real numbers in common ...
- 1
0
votes
1
answer
21
views
Binary crossentropy loss
When we have a binary classification problem, we use a sigmoid activation function in the output layer+ a binary crossentropy loss. We also need to one hot encode the target variable.This s a binary ...
- 221
0
votes
1
answer
67
views
Searching optimum ranges for maximizing a variable
I am currently working on a project where I have been asked to analyze a dataframe with various time series.
...
- 1
1
vote
1
answer
58
views
Calculate the top 5 optimal parcel locker cabinet configurations
Dear Data Science community,
I have the following problem to solve and I'd like to learn which algorithm or approach I can use to tackle it. I don't expect a full solution here but I really want to ...
- 111