Episode #125 of the Stack Overflow podcast is here. We talk Tilde Club and mechanical keyboards. Listen now

Questions tagged [theory]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
0answers
8 views

Evolution of classification methods

In the book "Deep Learning with Python" by Francois Chollet (2018), in section 1.2.4 one can find: Decisions trees learned from data began to receive significant research interest in the 2000s, and ...
1
vote
1answer
50 views

A question on realizable sample complexity

I came across the following exercise, and I just can't seem to crack it: Let $l$ be some loss function such that $l \leq 1$. Let $H$ be some hypothesis class, and let $A$ be a learning algorithm. ...
1
vote
0answers
26 views

What is the name of the prediction method?

Suppose, I have a data set: ix m_t1 m_t2 1 42 84 2 12 12 3 100 50 then, we can calculate the difference between ...
3
votes
1answer
27 views

Is it possible for a neural net to score as high as a different form of supervised learning?

I've been working with the Adult Census Income dataset from UCI http://archive.ics.uci.edu/ml/datasets/adult I've created two different models, one using a gradient boosted classifier with sklearn, ...
4
votes
1answer
68 views

A trick used in Rademacher complexity related Theorem

I am currently working on the proof of Theorem 3.1 in the book "Foundations of Machine Learning" (page 35, First edition), and there is a key trick used in the proof (equation 3.10 and 3.11): $$\...
5
votes
1answer
117 views

Are all classifiers linear in some high dimensional space?

Of all possible classifiers (including SVMs, locally weighted regression, softmax regression, lots others I'm sure I don't know about, etc.), are they all linear in some high dimensional space? E.g. ...
2
votes
1answer
33 views

An ambiguity in SVM equations about misclassified data

I have encountered an ambiguity in SVM equations. As is stated in Chris Bishop's machine learning book, the optimization goal in SVM is to maximize this function: $$C\sum\limits_{n = 1}^N {{\xi _n}} ...
0
votes
0answers
8 views

How to determine the rate of entry in a queue M/M/c?

I have the exit rate ($\mu$) and the average waiting time in the queue ($W_q$). I need solve to rate of input ($\lambda$) in a queue. I now: $\rho = \frac{\lambda}{c\mu} < 1$ $\pi_0 = \left[\left(...
0
votes
0answers
15 views

What could cause validation set to consistently perform better than training?

I'm training a neural network with a very small dataset just to get things set up, before training on a much larger set. (I only have about 500 data points available to me at this time, with more ...
0
votes
2answers
58 views

Decentralized machine learning

Currently, to train a model, you need to collect a huge blob of data. Are there feasible concepts of decentralized machine learning? Like, feed the model somehow from isolated data sources, or merge ...
0
votes
1answer
35 views

What does it mean when someone says “Most of the data science algorithms are optimization problems”

I was trying to understand the Gradient Descent algorithm from this article and the author says Most of the data science algorithms are optimization problems I come from software engineering ...
0
votes
1answer
41 views

What should be the requirement for training data in order to obtain a good regression model using neural network?

I have made a neural network regression model using the theory for the first time and would like to clarify some basic doubts, whose concrete answers I couldn't find yet. Data:- I have 3000 samples ...
1
vote
0answers
681 views

Candidate Elimination Algorithm - Simple Problem

I'm trying to understand version space learning and the Candidate Elimination algorithm. Define the set of most general and the set of most specific hypotheses. Take these training examples with the ...
9
votes
8answers
906 views

Difference between machine learning and artificial intelligence

My question is this: Is there any difference between machine learning and artificial intelligence? Or do these terms refer to the same thing?
1
vote
0answers
104 views

Intuition behind Occam's Learner Algorithm using VC-Dimension

So I'm learning about Occam's Learning algorithm and PAC-Learning where for a given hypothesis space $H$, if we want to have a model/hypothesis $h$ that has an True error of $error_D \leq \epsilon$, ...
1
vote
0answers
21 views

Is the hypothesis space spanned by kernel evaluations on datapoints equivalent to the hypothesis space of linear functionals in the feature space?

when studying kernel methods a few years ago I got a bit confused with the concepts of feature space, hypothesis space and reproducing kernel Hilbert space. Recently, I thought a little about ...
1
vote
0answers
61 views

How do you calculate the information capacity of a neural network?

Let's say I wanted to train a neural network to teach it the rules in a decision tree, so I generated a dataset by feeding arrays of random numbers into the pre-trained decision tree, and then used ...
4
votes
0answers
41 views

Learning a logical function with a 2 layer BDN network - manual weight setting rule question?

So I am trying to construct a 2-layer network of binary decision neurons as proposed by McCullough and Pitts (1943) to learn a logical function (a composition of AND's and OR's) such as: $((\neg x_1\...
0
votes
1answer
27 views

Situations where advanced theoretical knowledge of ML helped solve a real world problem?

I've invested lot of time trying to understand the theoretical aspects of Deep Learning and Neural Networks - but I'm now questioning whether it is worth it or not, given that I am someone who works ...
1
vote
0answers
30 views

Why study properties with infinitesimal change?

I read about analysis on local properties of neural networks. Some of them study the impact of "infinitesimal" change to an input. Like in Percy Liang's paper Understanding Black-box Predictions via ...
1
vote
1answer
76 views

What's the math for real world back-propagation?

Considering a simple ANN: $$x \rightarrow f=(U_{m\times n}x^T)^T \rightarrow g = g(f) \rightarrow h = (V_{p \times m}g^T)^T \rightarrow L = L(h,y) $$ where $x\in\mathbb{R}^n$, $U$ and $V$ are ...
2
votes
0answers
75 views

Is there a name for a scale which mixes ordinal and nominal?

The textbooks I have differentiate between nominal, ordinal, interval and ratio scales. The ordinal scale is quite popular in the wild, used for basically all subjective data, and also for dividing ...
1
vote
1answer
151 views

Softmax function forms

The constraint that the n outputs must sum to $1$ means that only $n−1$ parameters are necessary; the probability of the $n^{th}$ value may be obtained by subtracting the first $n−1$ probabilities ...
7
votes
1answer
993 views

Deriving backpropagation equations “natively” in tensor form

Image shows a typical layer somewhere in a feed forward network: $a_i^{(k)}$ is the activation value of the $i^{th}$ neuron in the $k^{th}$ layer. $W_{ij}^{(k)}$ is the weight connecting $i^{th}$ ...
1
vote
0answers
179 views

Global vs. local bias-variance tradeoff

In the standard example of decomposing the MSE into Bias, Variance and Irreducible error: $$MSE(x) = \left(\mathbb{E}[\hat{f}(x)] - f(x) \right)^2 + \mathbb{E}\left[\left(\hat{f}(x) - f(x)\right)^2\...
1
vote
3answers
1k views

What would be the best way to impute data?

Other than just filling in with the mean of a feature, what other methods are there which can work well? I am trying to decide whether or not to use a denoising-autoencoder or just impute with the ...