Questions tagged [theory]
Theory relates to theoretical questions regarding data science and machine learning.
61
questions
0
votes
0
answers
14
views
Reinforcement Learning: Formally, does an exponentially decaying epsilon satisfy GLIE?
I am aware that exponentially decaying exploration constants (epsilon) are used practically. Formally, though, do they satisfy the GLIE condition?
Specifically, relating this to the Borel-Cantelli ...
0
votes
0
answers
26
views
Check the validity of the distribution in the proof of No-Free-Lunch Theorem
I'm reading the proof of No-Free-Lunch Theorem (quoted at the end of this question) in Understanding Machine Learning: From Theory to Algorithms, Cambridge University Press, p.37, the author wrote:
...
- 155
0
votes
0
answers
13
views
Why do we use Gradient Descent for Norm Batch since the function has only 2 parameters?
So here is the formula for Z sedile ( gamma * Z + Beta). Since it's a function of 2 parameters why there is still stated in courses that we need to compute gradient descent to find the slope ? Since ...
0
votes
0
answers
20
views
What is an appropriate definition for the action space in my RL problem?
I am thinking about how to define the action space in my RL problem. The goal is to try many different RL algorithms (value-, policy-based, and hybrids), in order to compare their performance, across ...
1
vote
1
answer
59
views
Where does AI/ML theories come into play when nowadays the AI libraries already so powerful?
I've read up posts so-call for beginning ML, claiming you need linear algebra, statistics, complicated optimization to just getting start in ML/AI. And on top of these, there comes the ML/AI ...
- 411
1
vote
0
answers
26
views
Are there any established approaches for dealing with a Degenerate Feedback Loop?
Scenario:
I develop a model which forecasts the likely sales success of a particular enquiry based on outcomes of past similar enquiries.
I then assign this likelihood score to new enquiries when they ...
- 111
0
votes
1
answer
173
views
How to scale a subset of data with respect to the entire dataset
I am developing a financial time-series prediction model using sklearn using StandardScaler for scaling purposes. I train a model, and then use the model regularly ...
- 111
2
votes
0
answers
55
views
Minor error in Ian Goodfellow's GAN optimality proof
I've been thinking of a part of the proof of the optimality of GANs from the original paper, and I can't manage to solve what seems to be an error. The paper states that the maximum of the function $y ...
- 21
1
vote
1
answer
28
views
How do we derive our loss function from the gradient objective?
I've been dwelling through RL theory and practice and one particular part I find hard to properly understand is the relation between the practical loss function and ...
- 111
2
votes
0
answers
8
views
Learning the Average of a 0/1 Dependent Variable
uppose I have a matrix 𝑋 and a dependent vector 𝑦 whose entries are each
in {0,1}
dependent on the corresponding row of 𝑋
Given this dataset, I'd like to learn a model, so that given some other ...
- 1,227
1
vote
1
answer
36
views
Creating a map between N images and N labels using CNN
I have seen classification CNNs that train with numerous images for a subset of labels (i.e. Number of images >> Number of labels), however, is it still possible to use CNNs when the number of ...
- 11
0
votes
2
answers
58
views
Time series test data dilema
I’m trying to build a model to predict the amount of sales of a product for the next few days
This question is about whether or not I should use the tail of the serie as the test set and train models ...
- 1
1
vote
1
answer
100
views
Proof of GOSS algorithm in lightGBM paper
In the LightGBM paper the authors make use of a newly developed sampling method GOSS to reduce the number of data instances needed for finding the best split of a ...
- 121
1
vote
0
answers
1k
views
How to use the eval set in catboost appropriately?
Let's say you have a dataset, and you split it into 80% training and 20% testing. Naturally, you want to find the optimal hyperparameters for your model, so with the training set, you plan to do cross ...
- 11
1
vote
1
answer
18
views
Would all classification models perform similarly in a theoretical and ideal scenario?
Imagine that we count on infinite computation power, an infinite amount of data and we have an infinite amount of time to wait for a model to learn. In such a scenario, we want to have some data ...
- 243
0
votes
3
answers
171
views
Which neural network is better?
MNIST dataset with 60 000 training samples and 10 000 test samples.
Neural network #1. Accuracy on the training set: 99.53%. Accuracy on the test set: 99.31%.
Neural network #2. Accuracy on the ...
2
votes
3
answers
89
views
Why do the performance of DL models increase with the volume of data while that of ML models will flat out or even decrease?
I have read some articles and realized that many of them cited, for example, DLis better for large amount of data than ML.
Typically:
The performance of machine learning algorithms decreases as the ...
3
votes
1
answer
257
views
Given M binary variables and R samples, what is the maximum number of leaves in a decision tree?
Given M binary variables and R samples, what is the maximum number of leaves in a decision tree?
My first assumption was that the worst case would be a leaf for each sample, thus R leaves maximum. Am ...
- 33
2
votes
2
answers
89
views
How to input a list into my model and not have it care about order
I'm trying to predict a list of numbers, e.g:
[23,55,198,200,64]
The data I have includes multiple things, along with:
The numbers from the previous run (These ...
- 135
1
vote
1
answer
419
views
Use of multiple models vs training a single model for multiple outputs
So let's say I have data with numerical variables A, B and C.
I believe that the value of <...
- 135
3
votes
1
answer
376
views
What is Inductive bias?
Bias in a neural network is an additional neuron to be fired i.e let $y=a+bx$ where $a$ is a bias term.
Do we have any difference between bias and inductive bias?
How Inductive bias is helpful in ...
- 239
0
votes
1
answer
302
views
Multiple solutions with same minima in MLP with same weights
I came across an excercise on deep learning from here.
It goes as follows:
Consider a simple MLP with a single hidden layer of $d$ dimensions in the hidden layer and a single output. Show that for any ...
- 5
1
vote
1
answer
34
views
Theoretical basis for neural network "effort"
I might be in danger of having my question closed as "not clear what I'm asking for," but here goes.
Suppose we have a simple feedforward network. It has a few layers, each layer has a "...
- 111
1
vote
2
answers
550
views
What is the opposite of baseline?
I have created a prediction model and on the one hand I have to compare it with other baseline models, and on the other hand, I have to compare it with the ideal approach (supported by additional data)...
- 25
1
vote
1
answer
56
views
Are non-relu activations better for small/ dense datasets?
Building on the questions below, the only conclusion I could draw from the answers was that ReLu is less computationally expensive and better at sparsity.
Why is ...
- 519
0
votes
0
answers
18
views
Unbiased Predictions for all Distinct Training Subsets
Suppose I have a data set $\left(X_i \in \chi, y_i \in \zeta \right)$ where $X_i$ and $y_i$ correspond to instances and labels, and $\chi$ and $\zeta$ correspond to the space where $X_i$ and $y_i$ ...
- 101
2
votes
0
answers
233
views
How to get the maximum likelihood estimate of the categorical distribution parameters using Lagrange optimization?
Let's say our data is discrete-valued and belongs to one of $K$ classes.
The underlying probability distribution is assumed to be a categorical/multinoulli distribution given as $p(\textbf{x}) = \...
2
votes
0
answers
341
views
Why does Jim Gray call "data-driven science" a new paradigm?
Wikipedia it says about data science:
Turing award winner Jim Gray imagined data science as a "fourth paradigm" of science (empirical, theoretical, computational and now data-driven) and ...
- 752
1
vote
1
answer
167
views
Can a neural network be of variable depth?
It is very common for neural networks to be asymmetric about the x axis, that is, to have many more nuerons in the first few layers than in the last few layers. Common example:
But can neural ...
- 211
1
vote
1
answer
4k
views
What is an object detection problem with only one class called?
Object detection is defined as the problem in which a model needs to figure out the bounding boxes and the class for each object. A lot of ML solutions for object detection base around having "two ...
- 113
0
votes
1
answer
56
views
Gradient for starting Backpropagation
I was reading this nice tutorial about Pytorch's basics:
https://pytorch.org/tutorials/beginner/pytorch_with_examples.html
In the first example (pure Numpy), the author starts the backward phase by ...
- 103
1
vote
1
answer
101
views
What is the best approach for send time optimization? [closed]
I could no find a lot information about how the companies doabout send time optimization, either for push notifications or email campaigs. having historical data about clicks and sends what would be ...
- 11
1
vote
2
answers
568
views
Is the search space of Hyperparameters Continuous or Discrete?
I am looking into hyper-parameter tunning and was curious about whether the search space is considered continuous or discrete?
My understanding of both those cases:
1. Continuous would make it 'easier'...
- 11
0
votes
1
answer
34
views
What is the name of this statistical interaction?
What is the name of the following statistical / informational interaction:
given A, I know exactly what B is.
given B, I know to some extent what A is.
I'm not looking for a probability but rather ...
- 105
2
votes
1
answer
35
views
Structuring experiment/training data with months in mind
We're using a whole year's data to predict a certain target variable.The model works like data - OneHot encoding the categorical variables - MinMaxScaler - PCA (to choose a subset of 2000 components ...
- 1,310
2
votes
1
answer
104
views
Encoding correlation
I have rather theory-based question as I'm not that experienced in encoders, embeddings etc. Scientifically I'm mostly oriented around novel evolutionary model-based methods.
Let's assume we have ...
- 814
3
votes
0
answers
2k
views
Explanation of inductive bias of Candidate Elimination Algorithm
The definition of inductive bias says that
The inductive bias (also known as learning bias) of a learning algorithm is the set of assumptions that the learner uses to predict outputs given inputs ...
- 161
0
votes
1
answer
120
views
Understanding LSTM Keras implementation
So I understand what LSTM units are. But I have trouble understanding the implementation / function in Keras framework.
Let's say, I add a layer
...
- 63
2
votes
1
answer
62
views
Popular classification algorithms over time
In the book "Deep Learning with Python" by Francois Chollet (2018), in section 1.2.4 one can find:
Decisions trees learned from data began to receive significant research interest in the ...
- 211
3
votes
1
answer
40
views
Is it possible for a neural net to score as high as a different form of supervised learning?
I've been working with the Adult Census Income dataset from UCI
http://archive.ics.uci.edu/ml/datasets/adult
I've created two different models, one using a gradient boosted classifier with sklearn, ...
user71276
4
votes
1
answer
145
views
A trick used in Rademacher complexity related Theorem
I am currently working on the proof of Theorem 3.1 in the book "Foundations of Machine Learning" (page 35, First edition), and there is a key trick used in the proof (equation 3.10 and 3.11):
$$\...
- 161
2
votes
1
answer
50
views
An ambiguity in SVM equations about misclassified data
I have encountered an ambiguity in SVM equations.
As is stated in Chris Bishop's machine learning book, the optimization goal in SVM is to maximize this function:
$$C\sum\limits_{n = 1}^N {{\xi _n}} ...
- 1,237
2
votes
1
answer
84
views
A question on realizable sample complexity
I came across the following exercise, and I just can't seem to crack it:
Let $l$ be some loss function such that $l \leq 1$. Let $H$ be some hypothesis class, and let $A$ be a learning algorithm. ...
1
vote
0
answers
58
views
What could cause validation set to consistently perform better than training?
I'm training a neural network with a very small dataset just to get things set up, before training on a much larger set. (I only have about 500 data points available to me at this time, with more ...
- 213
0
votes
2
answers
115
views
Decentralized machine learning
Currently, to train a model, you need to collect a huge blob of data.
Are there feasible concepts of decentralized machine learning? Like, feed the model somehow from isolated data sources, or merge ...
- 381
0
votes
1
answer
66
views
What does it mean when someone says "Most of the data science algorithms are optimization problems"
I was trying to understand the Gradient Descent algorithm from this article and the author says
Most of the data science algorithms are optimization problems
I come from software engineering ...
0
votes
1
answer
110
views
What should be the requirement for training data in order to obtain a good regression model using neural network?
I have made a neural network regression model using the theory for the first time and would like to clarify some basic doubts, whose concrete answers I couldn't find yet.
Data:- I have 3000 samples ...
- 103
2
votes
0
answers
156
views
Intuition behind Occam's Learner Algorithm using VC-Dimension
So I'm learning about Occam's Learning algorithm and PAC-Learning where for a given hypothesis space $H$, if we want to have a model/hypothesis $h$ that has an True error of $error_D \leq \epsilon$, ...
- 101
1
vote
0
answers
32
views
Is the hypothesis space spanned by kernel evaluations on datapoints equivalent to the hypothesis space of linear functionals in the feature space?
when studying kernel methods a few years ago I got a bit confused with the concepts of feature space, hypothesis space and reproducing kernel Hilbert space. Recently, I thought a little about ...
- 11
5
votes
1
answer
464
views
Are all classifiers linear in some high dimensional space?
Of all possible classifiers (including SVMs, locally weighted regression, softmax regression, lots others I'm sure I don't know about, etc.), are they all linear in some high dimensional space?
E.g. ...
- 51