Questions tagged [bias]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
20
votes
5answers
9k views

Why underfitting is called high bias and overfitting is called high variance?

I have been using terms like underfitting/overfitting and bias-variance tradeoff for quite some while in data science discussions and I understand that underfitting is associated with high bias and ...
8
votes
1answer
817 views

Question on bias-variance tradeoff and means of optimization

So I was wondering how does one, for example, can best optimize the model they are trying to build when confronted with issues presented by high bias or high variance. Now, of course, you can play ...
5
votes
2answers
775 views

What are bias and variance in machine learning?

I am studying machine learning, and I have encountered the concept of bias and variance. I am a university student and in the slides of my professor, the bias is defined as: $bias = E[error_s(h)]-...
5
votes
2answers
534 views

Bagging vs Boosting, Bias vs Variance, Depth of trees

I understand the main principle of bagging and boosting for classification and regression trees. My doubts are about the optimization of the hyperparameters, especially the depth of the trees First ...
4
votes
2answers
140 views

Trade off between Bias and Variance [closed]

What are the best ideas or approaches to trade off between bias and variance in Machine Learning models.
3
votes
1answer
87 views

Bagging vs pasting in ensemble learning

This is a citation from "Hands-on machine learning with Scikit-Learn, Keras and TensorFlow" by Aurelien Geron: "Bootstrapping introduces a bit more diversity in the subsets that each predictor is ...
3
votes
1answer
463 views

Bias-variance tradeoff in practice (CNN)

I first trained a CNN on my dataset and got a loss plot that looks somewhat like this: Orange is training loss, blue is dev loss. As you can see, the training loss is lower than the dev loss, so I ...
3
votes
1answer
34 views

Whether add bias or not in a perceptron

In some places, perceptron is described as having added bias, while in some places, bias is not added. Which one is right for you?
3
votes
1answer
394 views

Is normalizing the validation set of time series a kind of look ahead bias?

Here's the data normalization process of a time series in a paper about stock prediction using LSTM: Split train and test set based on time (e.g. training set: 2001-2010, test set:2011-2012). This ...
3
votes
1answer
600 views

Correcting log-bias in the output of an XGB

I have previously worked with GAMs, where I was trying to do regression on a log-transformed variable. The log-transformation introduced a negative bias in the average of the predicted variable, and I ...
3
votes
1answer
287 views

Learning curve using micro F-score and macro F-score

I plotted the learning curves using micro and macro F-scores for a Multinomial Naive Bayes classifier. The first plot is made using micro F-score, and the second using macro F-score. I find it quite ...
2
votes
1answer
214 views

How does C have effects on bias and variance of a Support Vector Machine?

The minimization problem for SVM can be written as- $$\overset{\text{min}}{\theta} C\sum_{i = 1}^{m}{[y^icost_1(\theta^Tx^i) + (1-y^i)cost_0(\theta^Tx^i)]} + \frac12\sum_{j = 1}^n{\theta_j}^2$$ Now, ...
2
votes
4answers
1k views

Linear machine learning algorithms “often” have high bias/low variance?

In this blog, which explains the meaning of bias and variance in machine learning, there's a line under the heading "Bias-Variance Trade-Off" which says: Parametric or linear machine learning ...
2
votes
1answer
52 views

Which between random forest or extra tree is best in a unbalance dataset?

I have an unbalanced dataset, with 3 classes, with 60% of class 1, 38% of class 2, and 2% of class 3. I don't want to generate more examples of class 3, and I cannot get more examples of class 3. The ...
2
votes
1answer
45 views

Why is it okay to set the bias vector up with zeros, and not the weight matrices?

We do not initialize weight matrices with zeros because the symmetry isn’t broken during the backward pass, and subsequently in the parameter updating process. But it is safe to set the bias vector up ...
2
votes
1answer
225 views

Bias-variance tradeoff and the uncertainty principle

Bias variance tradeoff seems to behave like the uncertainty principle, is it just another name for the same principle?
2
votes
1answer
53 views

svm optimization problem

Suppose we have the dataset: {(3,1),(3-1),(6,1),(6,-1)} {(1,0),(0,1),(0,-1),(-1,0)} the first set represent the positive label, and de second the negative. I want manually find the support vectors, ...
2
votes
2answers
71 views

Predictive modeling when output affects future input

Assume I have a model which predicts the outcome of number of icecream sold in a store. The model is trained on data for the last 5 years while keep the last year as a validation set and has produced ...
2
votes
1answer
29 views

How to measure deviance resulting from different random seeds in machine learning?

I'm running an xgboost model to predict probabilities to a binary classification problem. Then I aggregate the results based on the Age variable (what is the aggregated risk of getting the sickness ...
2
votes
2answers
38 views

How much can bias decrease performance of the network at the beginnng of the training?

I am writing a custom framework and in it I'm trying to train a simple network to predict the addition function. The network: 1 hidden layer of 3 Neurons 1 output layer cost function used is ...
2
votes
0answers
77 views

Bias Formula in Machine Learning expanded using ground truth

Why is Bias calculated for $f(x)$? Shouldn't it be calculated for $Y$ (which is $f(x)$ + Noise $\epsilon$)? We are fitting our model to $Y$, So shouldn't we be calculating bias wrt to $Y$? Also, I ...
2
votes
1answer
202 views

LSTM regression bias increases when targets go close to 0

I've build a LSTM model for time series forecasting. Results are not bad, with a mean normalized error of 7%. However, this normalized bias shows a clear pattern: The closer to 0 the value to predict, ...
2
votes
0answers
23 views

When should the bias b be updated with weights w and when should it be updated seperately?

It seems in some Machine Learning models, the bias term $b$ is updated just like other weights $w_i, i=1...n$. For example, in Logistic Regression, using SGD, $b \ \text{or} \ w_0$ is updated with: $$...
1
vote
2answers
188 views

Why is there a trade-off between bias and variance in supervised learning? Why can't we have best of both worlds?

The bias-variance trade-off is like a law in machine learning. You cannot have the best of both worlds. What is it about supervised learning in machine learning that makes it impossible to satisfy the ...
1
vote
2answers
242 views

How is the equation for the relation between prediction error, bias, and variance defined?

I'm reading this article Understanding the BiasVariance Tradeoff. It mentioned: If we denote the variable we are trying to predict as $Y$ and our covariates as $X$, we may assume that there is a ...
1
vote
1answer
62 views

Model biased towards low frequency data?

Generally model gets biased towards data_samples/target whose frequency is high in training data set. Is it possible during training that model gets biased towards low frequency training data set.
1
vote
1answer
33 views

Training dataset decreasing in quality (Google data science blog)

I have a complex algorithm that decides when it should show customers of an only shop an ad on our website, after they log in, in hope that they will buy what is in the ad. We have no control what is ...
1
vote
1answer
15 views

Weights and bias' relative to preprocessed X

I am currently using sklearn scale to preprocess my X data before being put into a perceptron - mean/stddev so as to prevent the ...
1
vote
1answer
32 views

Model Selection using Bias Variance Trade Off

I have a Regression Model with Train MAPE as 6% and Test MAPE as 15%. This appears to me as a clear case of over fitting. But can I still use this model assuming 15% Error is not a bad number after-...
1
vote
1answer
14 views

Is there a quantitative way to determine if a class of algorithms tends produce low bias or low variance models?

I understand that some machine learning models tend to be low bias, whereas others tend to be low variance (source). As an example, a linear regression will tend to have low variance error and high ...
1
vote
1answer
29 views

Do non-parametric models always overfit without regularization?

Let's scope this to just classification. It's clear that if you fully grow out a decision tree with no regularization (e.g. max depth, pruning), it will overfit the training data and get full accuracy ...
1
vote
0answers
76 views

Derivative of Loss wrt bias term

I read this and have an ambiguity. I try to understand well how to calculate the derivative of Loss w.r.t to bias. In this question, we have this definition: ...
1
vote
0answers
299 views

How to calculate irreducible error using Bias and Variance for a given machine learning Model?

I am trying to calculate the Bias and Variance for a ML Model. $$ Err(x)=E[(Y−\hat f(x))^2] \\Err(x)=Bias^2+Variance+Irreducible\ Error $$ $\hat f(x)$ is our model $Y$ is the variable we are trying ...
1
vote
0answers
14 views

Measuring the bias of a machine learning model

How can we measure the bias of a machine learning model? Can we determine it by just calculating its performance estimates difference on the train data and test data? For example, if a model SVM ...
1
vote
0answers
335 views

Calculation of Neural network biases in backpropagation

While learning neural networks I've found a basic Python working example to play with. It has 3 input nodes, 4 nodes in a hidden layer, 1 output node. 5 data sets for training. The initial code is ...
1
vote
1answer
30 views

What is the defining Set in NLP

I am reading the paper Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings here is the pdf. On page 6, we read: ...
1
vote
2answers
111 views

bias variance decomposition for classification problem

It is given that: MSE = bias$^2$ + variance I can see the mathematical relationship between MSE, bias, and variance. However, how do we understand the mathematical intuition of bias and variance for ...
1
vote
0answers
108 views

Math behind, MSE = bias^2 + variance

Based on the deeplearningbook: $$MSE = E[(\theta_m^{-} - \theta)^2]$$ $$equals$$ $$Bias(\theta_m^{-})^2 + Var(\theta_m^{-})$$ where m is the number of samples ...
1
vote
0answers
299 views

Simulate Biased dataset Python [closed]

I am trying simulate data from a normal distribution but bias the sample by excluding all negative values and values divisible by 5 . To demonstrate the effect of bias . I will probably calculate ...
1
vote
1answer
318 views

Free parameters in logistic regression

When applying logistic regression, one is essentially applying the following function $1/(1 + e^{\beta x})$ to provide a decision boundary, where $\beta$ are a set of parameters that are learned by ...
1
vote
0answers
199 views

Bias of 1 in fully connected layers introduced dying relu problem

While implementing AlexNet (model-code), one of the thing I need to do was to initialize the biases of the convolutional layers and fully connected layers. Normally we initialize biases with 0s, but ...
0
votes
2answers
4k views

Relationship between train and test error

I have some specific questions for which I could not extract answers from books. Therefore, I ask for help here and shall be extremely grateful for an intuitive explanation if possible. In general, ...
0
votes
1answer
29 views

Bias variance tradeoff boosting (xgboost) vs random forest (randomized bagging) which to use when?

I was looking up differences between boosting an bagging and I see this quoted everywhere If the classifier is unstable (high variance), then we should apply Bagging. If the classifier is stable and ...
0
votes
1answer
24 views

Does training of neural networks follow the same order in each epoch?

Each epoch uses the weight from the end of the previous epoch(correct me if I am wrong). Is the updating of parameters after each batch always in the same order? To rephrase, are the batches always in ...
0
votes
1answer
25 views

build biased image dataset for emotion analysis

This is a pre-project question. I would like to find or build a biased dataset to demonstrate what happens if training data are biased (biased distributed ethnicity for exemple). I try this for the ...
0
votes
1answer
43 views

Should bias updates be porportional to overfitting?

According to questions on the internet, the bias is a learnable parameter, and there are different solutions to updating it, but I failed to find a concise methodology of correctly updating biases ...
0
votes
1answer
154 views

Maths question on mean squared error being dervied to bias and variance

I am reading a book and have difficulty in understanding the math on bias- variance tradeoff. Below is the section that I am having trouble with: Given a set of training samples $x_1, x_2, ..., x_n$...
0
votes
1answer
88 views

Can You Purposely Bias A Clustering Model?

We have a large amount (Billions) of high cardinality, mixed nominal & numerical data, and are performing some clustering on it as an experiment. There is a small subset of these data, however, ...
0
votes
0answers
20 views

how to know if there is a bias in data collection methods

I am collecting data for machine learning models I want to build for some application. I started with random sampling (just simply collecting 'recent' data) but I am not getting enough records of ...
0
votes
0answers
18 views

Effect of batch during prediction

During prediction (not training), is it normal to get different loss for different batch size? Worst case happens when I use batch_size=1 for test dataset. The prediction performance get pretty bad. ...