Questions tagged [theory]

Theory relates to theoretical questions regarding data science and machine learning.

Filter by
Sorted by
Tagged with
2 votes
0 answers
7 views

Learning the Average of a 0/1 Dependent Variable

uppose I have a matrix 𝑋 and a dependent vector 𝑦 whose entries are each in {0,1} dependent on the corresponding row of 𝑋 Given this dataset, I'd like to learn a model, so that given some other ...
user avatar
  • 1,147
0 votes
0 answers
6 views

Geometric Deep Learning - G-Smoothing operator on polynomials

(Note: My question resolves about a problem stated in the following lecture video: https://youtu.be/ERL17gbbSwo?t=413 Hi, I hope this is the right forum for these kind of questions. I'm currently ...
user avatar
  • 1
1 vote
1 answer
10 views

Creating a map between N images and N labels using CNN

I have seen classification CNNs that train with numerous images for a subset of labels (i.e. Number of images >> Number of labels), however, is it still possible to use CNNs when the number of ...
user avatar
  • 11
0 votes
2 answers
46 views

Time series test data dilema

I’m trying to build a model to predict the amount of sales of a product for the next few days This question is about whether or not I should use the tail of the serie as the test set and train models ...
user avatar
0 votes
1 answer
20 views

Proof of GOSS algorithm in lightGBM paper

In the LightGBM paper the authors make use of a newly developed sampling method GOSS to reduce the number of data instances needed for finding the best split of a ...
user avatar
  • 101
0 votes
0 answers
26 views

Lasso (or Ridge) vs Bayesian MAP

This is the first time I have posted here. I am looking for some feedback or perspective on this question. To make it simple, let's just talk about linear models. We know the MLE solution for the $l_1$...
user avatar
0 votes
0 answers
174 views

Can XGBoost support vector outputs?

I am interested in fitting data (regression rather than classification) with individual targets which are vectors via an XGBoost type model. However, currently Python's xgboost.XGBRegressor model only ...
user avatar
  • 1
0 votes
0 answers
280 views

How to use the eval set in catboost appropriately?

Let's say you have a dataset, and you split it into 80% training and 20% testing. Naturally, you want to find the optimal hyperparameters for your model, so with the training set, you plan to do cross ...
user avatar
0 votes
0 answers
18 views

End-to-end machine learning project processes

I've read a book chapter that walks you through all the steps involved in an end-to-end machine learning project. After doing all the practical exercises I'm still not quite sure that my way of ...
user avatar
1 vote
1 answer
14 views

Would all classification models perform similarly in a theoretical and ideal scenario?

Imagine that we count on infinite computation power, an infinite amount of data and we have an infinite amount of time to wait for a model to learn. In such a scenario, we want to have some data ...
user avatar
  • 243
0 votes
3 answers
105 views

Which neural network is better?

MNIST dataset with 60 000 training samples and 10 000 test samples. Neural network #1. Accuracy on the training set: 99.53%. Accuracy on the test set: 99.31%. Neural network #2. Accuracy on the ...
user avatar
2 votes
2 answers
46 views

Why do the performance of DL models increase with the volume of data while that of ML models will flat out or even decrease?

I have read some articles and realized that many of them cited, for example, DLis better for large amount of data than ML. Typically: The performance of machine learning algorithms decreases as the ...
user avatar
3 votes
1 answer
80 views

Given M binary variables and R samples, what is the maximum number of leaves in a decision tree?

Given M binary variables and R samples, what is the maximum number of leaves in a decision tree? My first assumption was that the worst case would be a leaf for each sample, thus R leaves maximum. Am ...
user avatar
2 votes
2 answers
29 views

How to input a list into my model and not have it care about order

I'm trying to predict a list of numbers, e.g: [23,55,198,200,64] The data I have includes multiple things, along with: The numbers from the previous run (These ...
user avatar
0 votes
1 answer
115 views

Use of multiple models vs training a single model for multiple outputs

So let's say I have data with numerical variables A, B and C. I believe that the value of a has an effect on B. I also believe that A and B both have an effect on C. I don't think C has an effect on ...
user avatar
0 votes
0 answers
9 views

How to introduce a parameter for measuring change in data over time

In my project, I need to introduce a measure for 'movement' using a 3axis accelerometer (ADXL345). As sketched below: I thought about introducing some micro-changes, i.e. absolute change in ...
user avatar
3 votes
1 answer
280 views

What is Inductive bias?

Bias in a neural network is an additional neuron to be fired i.e let $y=a+bx$ where $a$ is a bias term. Do we have any difference between bias and inductive bias? How Inductive bias is helpful in ...
user avatar
0 votes
1 answer
85 views

Multiple solutions with same minima in MLP with same weights

I came across an excercise on deep learning from here. It goes as follows: Consider a simple MLP with a single hidden layer of $d$ dimensions in the hidden layer and a single output. Show that for any ...
user avatar
1 vote
1 answer
29 views

Theoretical basis for neural network "effort"

I might be in danger of having my question closed as "not clear what I'm asking for," but here goes. Suppose we have a simple feedforward network. It has a few layers, each layer has a "...
user avatar
1 vote
2 answers
220 views

What is the opposite of baseline?

I have created a prediction model and on the one hand I have to compare it with other baseline models, and on the other hand, I have to compare it with the ideal approach (supported by additional data)...
user avatar
1 vote
1 answer
25 views

Are non-relu activations better for small/ dense datasets?

Building on the questions below, the only conclusion I could draw from the answers was that ReLu is less computationally expensive and better at sparsity. Why is ...
user avatar
  • 476
0 votes
0 answers
17 views

Unbiased Predictions for all Distinct Training Subsets

Suppose I have a data set $\left(X_i \in \chi, y_i \in \zeta \right)$ where $X_i$ and $y_i$ correspond to instances and labels, and $\chi$ and $\zeta$ correspond to the space where $X_i$ and $y_i$ ...
user avatar
2 votes
0 answers
130 views

How to get the maximum likelihood estimate of the categorical distribution parameters using Lagrange optimization?

Let's say our data is discrete-valued and belongs to one of $K$ classes. The underlying probability distribution is assumed to be a categorical/multinoulli distribution given as $p(\textbf{x}) = \...
user avatar
2 votes
0 answers
208 views

Why does Jim Gray call "data-driven science" a new paradigm?

Wikipedia it says about data science: Turing award winner Jim Gray imagined data science as a "fourth paradigm" of science (empirical, theoretical, computational and now data-driven) and ...
user avatar
  • 732
1 vote
1 answer
109 views

Can a neural network be of variable depth?

It is very common for neural networks to be asymmetric about the x axis, that is, to have many more nuerons in the first few layers than in the last few layers. Common example: But can neural ...
user avatar
  • 211
1 vote
1 answer
2k views

What is an object detection problem with only one class called?

Object detection is defined as the problem in which a model needs to figure out the bounding boxes and the class for each object. A lot of ML solutions for object detection base around having "two ...
user avatar
  • 113
0 votes
1 answer
46 views

Gradient for starting Backpropagation

I was reading this nice tutorial about Pytorch's basics: https://pytorch.org/tutorials/beginner/pytorch_with_examples.html In the first example (pure Numpy), the author starts the backward phase by ...
user avatar
  • 103
1 vote
1 answer
58 views

What is the best approach for send time optimization? [closed]

I could no find a lot information about how the companies doabout send time optimization, either for push notifications or email campaigs. having historical data about clicks and sends what would be ...
user avatar
1 vote
2 answers
300 views

Is the search space of Hyperparameters Continuous or Discrete?

I am looking into hyper-parameter tunning and was curious about whether the search space is considered continuous or discrete? My understanding of both those cases: 1. Continuous would make it 'easier'...
user avatar
0 votes
1 answer
30 views

What is the name of this statistical interaction?

What is the name of the following statistical / informational interaction: given A, I know exactly what B is. given B, I know to some extent what A is. I'm not looking for a probability but rather ...
user avatar
  • 105
2 votes
1 answer
30 views

Structuring experiment/training data with months in mind

We're using a whole year's data to predict a certain target variable.The model works like data - OneHot encoding the categorical variables - MinMaxScaler - PCA (to choose a subset of 2000 components ...
user avatar
  • 1,248
2 votes
1 answer
67 views

Encoding correlation

I have rather theory-based question as I'm not that experienced in encoders, embeddings etc. Scientifically I'm mostly oriented around novel evolutionary model-based methods. Let's assume we have ...
user avatar
2 votes
0 answers
1k views

Explanation of inductive bias of Candidate Elimination Algorithm

The definition of inductive bias says that The inductive bias (also known as learning bias) of a learning algorithm is the set of assumptions that the learner uses to predict outputs given inputs ...
user avatar
0 votes
1 answer
92 views

Understanding LSTM Keras implementation

So I understand what LSTM units are. But I have trouble understanding the implementation / function in Keras framework. Let's say, I add a layer ...
user avatar
2 votes
1 answer
52 views

Popular classification algorithms over time

In the book "Deep Learning with Python" by Francois Chollet (2018), in section 1.2.4 one can find: Decisions trees learned from data began to receive significant research interest in the ...
user avatar
3 votes
1 answer
37 views

Is it possible for a neural net to score as high as a different form of supervised learning?

I've been working with the Adult Census Income dataset from UCI http://archive.ics.uci.edu/ml/datasets/adult I've created two different models, one using a gradient boosted classifier with sklearn, ...
user avatar
4 votes
1 answer
129 views

A trick used in Rademacher complexity related Theorem

I am currently working on the proof of Theorem 3.1 in the book "Foundations of Machine Learning" (page 35, First edition), and there is a key trick used in the proof (equation 3.10 and 3.11): $$\...
user avatar
2 votes
1 answer
42 views

An ambiguity in SVM equations about misclassified data

I have encountered an ambiguity in SVM equations. As is stated in Chris Bishop's machine learning book, the optimization goal in SVM is to maximize this function: $$C\sum\limits_{n = 1}^N {{\xi _n}} ...
user avatar
  • 1,207
2 votes
1 answer
78 views

A question on realizable sample complexity

I came across the following exercise, and I just can't seem to crack it: Let $l$ be some loss function such that $l \leq 1$. Let $H$ be some hypothesis class, and let $A$ be a learning algorithm. ...
user avatar
1 vote
0 answers
57 views

What could cause validation set to consistently perform better than training?

I'm training a neural network with a very small dataset just to get things set up, before training on a much larger set. (I only have about 500 data points available to me at this time, with more ...
user avatar
0 votes
2 answers
98 views

Decentralized machine learning

Currently, to train a model, you need to collect a huge blob of data. Are there feasible concepts of decentralized machine learning? Like, feed the model somehow from isolated data sources, or merge ...
user avatar
0 votes
1 answer
52 views

What does it mean when someone says "Most of the data science algorithms are optimization problems"

I was trying to understand the Gradient Descent algorithm from this article and the author says Most of the data science algorithms are optimization problems I come from software engineering ...
user avatar
0 votes
1 answer
98 views

What should be the requirement for training data in order to obtain a good regression model using neural network?

I have made a neural network regression model using the theory for the first time and would like to clarify some basic doubts, whose concrete answers I couldn't find yet. Data:- I have 3000 samples ...
user avatar
  • 103
2 votes
0 answers
153 views

Intuition behind Occam's Learner Algorithm using VC-Dimension

So I'm learning about Occam's Learning algorithm and PAC-Learning where for a given hypothesis space $H$, if we want to have a model/hypothesis $h$ that has an True error of $error_D \leq \epsilon$, ...
user avatar
1 vote
0 answers
29 views

Is the hypothesis space spanned by kernel evaluations on datapoints equivalent to the hypothesis space of linear functionals in the feature space?

when studying kernel methods a few years ago I got a bit confused with the concepts of feature space, hypothesis space and reproducing kernel Hilbert space. Recently, I thought a little about ...
user avatar
  • 11
5 votes
1 answer
342 views

Are all classifiers linear in some high dimensional space?

Of all possible classifiers (including SVMs, locally weighted regression, softmax regression, lots others I'm sure I don't know about, etc.), are they all linear in some high dimensional space? E.g. ...
user avatar
  • 51
1 vote
0 answers
100 views

How do you calculate the information capacity of a neural network?

Let's say I wanted to train a neural network to teach it the rules in a decision tree, so I generated a dataset by feeding arrays of random numbers into the pre-trained decision tree, and then used ...
user avatar
5 votes
0 answers
103 views

Learning a logical function with a 2 layer BDN network - manual weight setting rule question?

So I am trying to construct a 2-layer network of binary decision neurons as proposed by McCullough and Pitts (1943) to learn a logical function (a composition of AND's and OR's) such as: $((\neg x_1\...
user avatar
0 votes
1 answer
41 views

Situations where advanced theoretical knowledge of ML helped solve a real world problem?

I've invested lot of time trying to understand the theoretical aspects of Deep Learning and Neural Networks - but I'm now questioning whether it is worth it or not, given that I am someone who works ...
user avatar
1 vote
0 answers
33 views

Why study properties with infinitesimal change?

I read about analysis on local properties of neural networks. Some of them study the impact of "infinitesimal" change to an input. Like in Percy Liang's paper Understanding Black-box Predictions via ...
user avatar
  • 11