Questions tagged [theory]

Theory relates to theoretical questions regarding data science and machine learning.

Filter by
Sorted by
Tagged with
0
votes
0answers
28 views

Can XGBoost support vector outputs?

I am interested in fitting data (regression rather than classification) with individual targets which are vectors via an XGBoost type model. However, currently Python's xgboost.XGBRegressor model only ...
0
votes
0answers
23 views

How to use the eval set in catboost appropriately?

Let's say you have a dataset, and you split it into 80% training and 20% testing. Naturally, you want to find the optimal hyperparameters for your model, so with the training set, you plan to do cross ...
0
votes
0answers
14 views

End-to-end machine learning project processes

I've read a book chapter that walks you through all the steps involved in an end-to-end machine learning project. After doing all the practical exercises I'm still not quite sure that my way of ...
1
vote
1answer
14 views

Would all classification models perform similarly in a theoretical and ideal scenario?

Imagine that we count on infinite computation power, an infinite amount of data and we have an infinite amount of time to wait for a model to learn. In such a scenario, we want to have some data ...
0
votes
3answers
62 views

Which neural network is better?

MNIST dataset with 60 000 training samples and 10 000 test samples. Neural network #1. Accuracy on the training set: 99.53%. Accuracy on the test set: 99.31%. Neural network #2. Accuracy on the ...
2
votes
1answer
33 views

Why do the performance of DL models increase with the volume of data while that of ML models will flat out or even decrease?

I have read some articles and realized that many of them cited, for example, DLis better for large amount of data than ML. Typically: The performance of machine learning algorithms decreases as the ...
3
votes
1answer
53 views

Given M binary variables and R samples, what is the maximum number of leaves in a decision tree?

Given M binary variables and R samples, what is the maximum number of leaves in a decision tree? My first assumption was that the worst case would be a leaf for each sample, thus R leaves maximum. Am ...
2
votes
2answers
26 views

How to input a list into my model and not have it care about order

I'm trying to predict a list of numbers, e.g: [23,55,198,200,64] The data I have includes multiple things, along with: The numbers from the previous run (These ...
0
votes
1answer
31 views

Use of multiple models vs training a single model for multiple outputs

So let's say I have data with numerical variables A, B and C. I believe that the value of a has an effect on B. I also believe that A and B both have an effect on C. I don't think C has an effect on ...
0
votes
0answers
8 views

How to introduce a parameter for measuring change in data over time

In my project, I need to introduce a measure for 'movement' using a 3axis accelerometer (ADXL345). As sketched below: I thought about introducing some micro-changes, i.e. absolute change in ...
3
votes
1answer
229 views

What is Inductive bias?

Bias in a neural network is an additional neuron to be fired i.e let $y=a+bx$ where $a$ is a bias term. Do we have any difference between bias and inductive bias? How Inductive bias is helpful in ...
0
votes
1answer
36 views

Multiple solutions with same minima in MLP with same weights

I came across an excercise on deep learning from here. It goes as follows: Consider a simple MLP with a single hidden layer of $d$ dimensions in the hidden layer and a single output. Show that for any ...
1
vote
1answer
24 views

Theoretical basis for neural network "effort"

I might be in danger of having my question closed as "not clear what I'm asking for," but here goes. Suppose we have a simple feedforward network. It has a few layers, each layer has a "...
1
vote
2answers
134 views

What is the opposite of baseline?

I have created a prediction model and on the one hand I have to compare it with other baseline models, and on the other hand, I have to compare it with the ideal approach (supported by additional data)...
1
vote
1answer
22 views

Are non-relu activations better for small/ dense datasets?

Building on the questions below, the only conclusion I could draw from the answers was that ReLu is less computationally expensive and better at sparsity. Why is ...
0
votes
0answers
13 views

Skipgram model theory confusion

In the output layer of a skipgram model, there are $|\text{Context}|*|\text{Vocab}|$ values. And for each context word, the values are basically the dot product of the input word representation and ...
0
votes
0answers
17 views

Unbiased Predictions for all Distinct Training Subsets

Suppose I have a data set $\left(X_i \in \chi, y_i \in \zeta \right)$ where $X_i$ and $y_i$ correspond to instances and labels, and $\chi$ and $\zeta$ correspond to the space where $X_i$ and $y_i$ ...
0
votes
0answers
11 views

Theoretically Speaking, How Do Squeeze-and-Excitation Blocks Help?

A SE block works by assigning a weight to each channel, contrary to a vanilla filter, which gives equal importance to all channels. My question is, theoretically speaking, shouldn't a regular filter ...
2
votes
0answers
164 views

Why does Jim Gray call "data-driven science" a new paradigm?

Wikipedia it says about data science: Turing award winner Jim Gray imagined data science as a "fourth paradigm" of science (empirical, theoretical, computational and now data-driven) and ...
1
vote
1answer
88 views

Can a neural network be of variable depth?

It is very common for neural networks to be asymmetric about the x axis, that is, to have many more nuerons in the first few layers than in the last few layers. Common example: But can neural ...
1
vote
1answer
750 views

What is an object detection problem with only one class called?

Object detection is defined as the problem in which a model needs to figure out the bounding boxes and the class for each object. A lot of ML solutions for object detection base around having "two ...
0
votes
1answer
45 views

Gradient for starting Backpropagation

I was reading this nice tutorial about Pytorch's basics: https://pytorch.org/tutorials/beginner/pytorch_with_examples.html In the first example (pure Numpy), the author starts the backward phase by ...
1
vote
1answer
43 views

What is the best approach for send time optimization? [closed]

I could no find a lot information about how the companies doabout send time optimization, either for push notifications or email campaigs. having historical data about clicks and sends what would be ...
1
vote
2answers
234 views

Is the search space of Hyperparameters Continuous or Discrete?

I am looking into hyper-parameter tunning and was curious about whether the search space is considered continuous or discrete? My understanding of both those cases: 1. Continuous would make it 'easier'...
0
votes
1answer
30 views

What is the name of this statistical interaction?

What is the name of the following statistical / informational interaction: given A, I know exactly what B is. given B, I know to some extent what A is. I'm not looking for a probability but rather ...
2
votes
1answer
30 views

Structuring experiment/training data with months in mind

We're using a whole year's data to predict a certain target variable.The model works like data - OneHot encoding the categorical variables - MinMaxScaler - PCA (to choose a subset of 2000 components ...
2
votes
1answer
48 views

Encoding correlation

I have rather theory-based question as I'm not that experienced in encoders, embeddings etc. Scientifically I'm mostly oriented around novel evolutionary model-based methods. Let's assume we have ...
0
votes
1answer
81 views

Understanding LSTM Keras implementation

So I understand what LSTM units are. But I have trouble understanding the implementation / function in Keras framework. Let's say, I add a layer ...
2
votes
1answer
48 views

Popular classification algorithms over time

In the book "Deep Learning with Python" by Francois Chollet (2018), in section 1.2.4 one can find: Decisions trees learned from data began to receive significant research interest in the ...
3
votes
1answer
32 views

Is it possible for a neural net to score as high as a different form of supervised learning?

I've been working with the Adult Census Income dataset from UCI http://archive.ics.uci.edu/ml/datasets/adult I've created two different models, one using a gradient boosted classifier with sklearn, ...
4
votes
1answer
110 views

A trick used in Rademacher complexity related Theorem

I am currently working on the proof of Theorem 3.1 in the book "Foundations of Machine Learning" (page 35, First edition), and there is a key trick used in the proof (equation 3.10 and 3.11): $$\...
2
votes
1answer
41 views

An ambiguity in SVM equations about misclassified data

I have encountered an ambiguity in SVM equations. As is stated in Chris Bishop's machine learning book, the optimization goal in SVM is to maximize this function: $$C\sum\limits_{n = 1}^N {{\xi _n}} ...
2
votes
1answer
74 views

A question on realizable sample complexity

I came across the following exercise, and I just can't seem to crack it: Let $l$ be some loss function such that $l \leq 1$. Let $H$ be some hypothesis class, and let $A$ be a learning algorithm. ...
1
vote
0answers
39 views

What could cause validation set to consistently perform better than training?

I'm training a neural network with a very small dataset just to get things set up, before training on a much larger set. (I only have about 500 data points available to me at this time, with more ...
0
votes
2answers
95 views

Decentralized machine learning

Currently, to train a model, you need to collect a huge blob of data. Are there feasible concepts of decentralized machine learning? Like, feed the model somehow from isolated data sources, or merge ...
0
votes
1answer
47 views

What does it mean when someone says "Most of the data science algorithms are optimization problems"

I was trying to understand the Gradient Descent algorithm from this article and the author says Most of the data science algorithms are optimization problems I come from software engineering ...
0
votes
1answer
89 views

What should be the requirement for training data in order to obtain a good regression model using neural network?

I have made a neural network regression model using the theory for the first time and would like to clarify some basic doubts, whose concrete answers I couldn't find yet. Data:- I have 3000 samples ...
2
votes
0answers
146 views

Intuition behind Occam's Learner Algorithm using VC-Dimension

So I'm learning about Occam's Learning algorithm and PAC-Learning where for a given hypothesis space $H$, if we want to have a model/hypothesis $h$ that has an True error of $error_D \leq \epsilon$, ...
1
vote
0answers
26 views

Is the hypothesis space spanned by kernel evaluations on datapoints equivalent to the hypothesis space of linear functionals in the feature space?

when studying kernel methods a few years ago I got a bit confused with the concepts of feature space, hypothesis space and reproducing kernel Hilbert space. Recently, I thought a little about ...
5
votes
1answer
287 views

Are all classifiers linear in some high dimensional space?

Of all possible classifiers (including SVMs, locally weighted regression, softmax regression, lots others I'm sure I don't know about, etc.), are they all linear in some high dimensional space? E.g. ...
1
vote
0answers
95 views

How do you calculate the information capacity of a neural network?

Let's say I wanted to train a neural network to teach it the rules in a decision tree, so I generated a dataset by feeding arrays of random numbers into the pre-trained decision tree, and then used ...
4
votes
0answers
80 views

Learning a logical function with a 2 layer BDN network - manual weight setting rule question?

So I am trying to construct a 2-layer network of binary decision neurons as proposed by McCullough and Pitts (1943) to learn a logical function (a composition of AND's and OR's) such as: $((\neg x_1\...
0
votes
1answer
38 views

Situations where advanced theoretical knowledge of ML helped solve a real world problem?

I've invested lot of time trying to understand the theoretical aspects of Deep Learning and Neural Networks - but I'm now questioning whether it is worth it or not, given that I am someone who works ...
1
vote
0answers
32 views

Why study properties with infinitesimal change?

I read about analysis on local properties of neural networks. Some of them study the impact of "infinitesimal" change to an input. Like in Percy Liang's paper Understanding Black-box Predictions via ...
1
vote
1answer
86 views

What's the math for real world back-propagation?

Considering a simple ANN: $$x \rightarrow f=(U_{m\times n}x^T)^T \rightarrow g = g(f) \rightarrow h = (V_{p \times m}g^T)^T \rightarrow L = L(h,y) $$ where $x\in\mathbb{R}^n$, $U$ and $V$ are ...
3
votes
1answer
101 views

Is there a name for a scale which mixes ordinal and nominal?

The textbooks I have differentiate between nominal, ordinal, interval and ratio scales. The ordinal scale is quite popular in the wild, used for basically all subjective data, and also for dividing ...
1
vote
1answer
200 views

Softmax function forms

The constraint that the n outputs must sum to $1$ means that only $n−1$ parameters are necessary; the probability of the $n^{th}$ value may be obtained by subtracting the first $n−1$ probabilities ...
13
votes
8answers
1k views

Difference between machine learning and artificial intelligence

Is there any difference between machine learning and artificial intelligence? Or do these terms refer to the same thing?
8
votes
1answer
1k views

Deriving backpropagation equations "natively" in tensor form

Image shows a typical layer somewhere in a feed forward network: $a_i^{(k)}$ is the activation value of the $i^{th}$ neuron in the $k^{th}$ layer. $W_{ij}^{(k)}$ is the weight connecting $i^{th}$ ...
1
vote
0answers
183 views

Global vs. local bias-variance tradeoff

In the standard example of decomposing the MSE into Bias, Variance and Irreducible error: $$MSE(x) = \left(\mathbb{E}[\hat{f}(x)] - f(x) \right)^2 + \mathbb{E}\left[\left(\hat{f}(x) - f(x)\right)^2\...