# Questions tagged [theory]

Theory relates to theoretical questions regarding data science and machine learning.

57 questions
Filter by
Sorted by
Tagged with
7 views

### Learning the Average of a 0/1 Dependent Variable

uppose I have a matrix 𝑋 and a dependent vector 𝑦 whose entries are each in {0,1} dependent on the corresponding row of 𝑋 Given this dataset, I'd like to learn a model, so that given some other ...
• 1,147
6 views

### Geometric Deep Learning - G-Smoothing operator on polynomials

(Note: My question resolves about a problem stated in the following lecture video: https://youtu.be/ERL17gbbSwo?t=413 Hi, I hope this is the right forum for these kind of questions. I'm currently ...
1 vote
10 views

### Creating a map between N images and N labels using CNN

I have seen classification CNNs that train with numerous images for a subset of labels (i.e. Number of images >> Number of labels), however, is it still possible to use CNNs when the number of ...
• 11
46 views

### Time series test data dilema

I’m trying to build a model to predict the amount of sales of a product for the next few days This question is about whether or not I should use the tail of the serie as the test set and train models ...
20 views

### Proof of GOSS algorithm in lightGBM paper

In the LightGBM paper the authors make use of a newly developed sampling method GOSS to reduce the number of data instances needed for finding the best split of a ...
• 101
26 views

### Lasso (or Ridge) vs Bayesian MAP

This is the first time I have posted here. I am looking for some feedback or perspective on this question. To make it simple, let's just talk about linear models. We know the MLE solution for the $l_1$...
174 views

### Can XGBoost support vector outputs?

I am interested in fitting data (regression rather than classification) with individual targets which are vectors via an XGBoost type model. However, currently Python's xgboost.XGBRegressor model only ...
280 views

### How to use the eval set in catboost appropriately?

Let's say you have a dataset, and you split it into 80% training and 20% testing. Naturally, you want to find the optimal hyperparameters for your model, so with the training set, you plan to do cross ...
18 views

### End-to-end machine learning project processes

I've read a book chapter that walks you through all the steps involved in an end-to-end machine learning project. After doing all the practical exercises I'm still not quite sure that my way of ...
1 vote
14 views

### Would all classification models perform similarly in a theoretical and ideal scenario?

Imagine that we count on infinite computation power, an infinite amount of data and we have an infinite amount of time to wait for a model to learn. In such a scenario, we want to have some data ...
• 243
105 views

### Which neural network is better?

MNIST dataset with 60 000 training samples and 10 000 test samples. Neural network #1. Accuracy on the training set: 99.53%. Accuracy on the test set: 99.31%. Neural network #2. Accuracy on the ...
46 views

### Why do the performance of DL models increase with the volume of data while that of ML models will flat out or even decrease?

I have read some articles and realized that many of them cited, for example, DLis better for large amount of data than ML. Typically: The performance of machine learning algorithms decreases as the ...
80 views

### Given M binary variables and R samples, what is the maximum number of leaves in a decision tree?

Given M binary variables and R samples, what is the maximum number of leaves in a decision tree? My first assumption was that the worst case would be a leaf for each sample, thus R leaves maximum. Am ...
• 33
29 views

### How to input a list into my model and not have it care about order

I'm trying to predict a list of numbers, e.g: [23,55,198,200,64] The data I have includes multiple things, along with: The numbers from the previous run (These ...
115 views

### Use of multiple models vs training a single model for multiple outputs

So let's say I have data with numerical variables A, B and C. I believe that the value of a has an effect on B. I also believe that A and B both have an effect on C. I don't think C has an effect on ...
9 views

### How to introduce a parameter for measuring change in data over time

In my project, I need to introduce a measure for 'movement' using a 3axis accelerometer (ADXL345). As sketched below: I thought about introducing some micro-changes, i.e. absolute change in ...
280 views

### What is Inductive bias?

Bias in a neural network is an additional neuron to be fired i.e let $y=a+bx$ where $a$ is a bias term. Do we have any difference between bias and inductive bias? How Inductive bias is helpful in ...
85 views

### Multiple solutions with same minima in MLP with same weights

I came across an excercise on deep learning from here. It goes as follows: Consider a simple MLP with a single hidden layer of $d$ dimensions in the hidden layer and a single output. Show that for any ...
1 vote
29 views

### Theoretical basis for neural network "effort"

I might be in danger of having my question closed as "not clear what I'm asking for," but here goes. Suppose we have a simple feedforward network. It has a few layers, each layer has a "...
• 111
1 vote
220 views

### What is the opposite of baseline?

I have created a prediction model and on the one hand I have to compare it with other baseline models, and on the other hand, I have to compare it with the ideal approach (supported by additional data)...
• 25
1 vote
25 views

### Are non-relu activations better for small/ dense datasets?

Building on the questions below, the only conclusion I could draw from the answers was that ReLu is less computationally expensive and better at sparsity. Why is ...
• 476
17 views

### Unbiased Predictions for all Distinct Training Subsets

Suppose I have a data set $\left(X_i \in \chi, y_i \in \zeta \right)$ where $X_i$ and $y_i$ correspond to instances and labels, and $\chi$ and $\zeta$ correspond to the space where $X_i$ and $y_i$ ...
• 101
130 views

41 views

### Situations where advanced theoretical knowledge of ML helped solve a real world problem?

I've invested lot of time trying to understand the theoretical aspects of Deep Learning and Neural Networks - but I'm now questioning whether it is worth it or not, given that I am someone who works ...
1 vote
33 views

### Why study properties with infinitesimal change?

I read about analysis on local properties of neural networks. Some of them study the impact of "infinitesimal" change to an input. Like in Percy Liang's paper Understanding Black-box Predictions via ...
• 11