Hot answers tagged

23 votes

Why underfitting is called high bias and overfitting is called high variance?

How can one understand it intuitively? Underfitting is called "Simplifying assumption" (Model is HIGHLY BIASED towards its assumption). your model will think linear hyperplane is good enough to ...
10 votes
Accepted

Question on bias-variance tradeoff and means of optimization

There are a lot of ways bias and variance can be minimized and despite the popular saying it isn't always a tradeoff. The two main reasons for high bias are insufficient model capacity and ...
  • 7,618
10 votes
Accepted

What are bias and variance in machine learning?

What are Bias and Variance? Let's start with some basic definitions: Bias: it's the difference between average predictions and true values. Variance: it's the variability of our predictions, i.e. how ...
  • 5,697
5 votes

Why underfitting is called high bias and overfitting is called high variance?

Let us assume our model to be described by $y = f(x) +\epsilon$, with $E[\epsilon]=0, \sigma_{\epsilon}\neq 0$. Let furthermore $\hat{f}(x)$ be our regression function, i.e. the function whose ...
  • 546
5 votes
Accepted

Why is there a trade-off between bias and variance in supervised learning? Why can't we have best of both worlds?

The tradeoff between bias and variance summarizes the "tug of war" game between fitting a model that predicts the underlying training dataset well (low bias) and producing a model that doesn't change ...
  • 1,834
5 votes

Bagging vs Boosting, Bias vs Variance, Depth of trees

Question 1: Bagging (Random Forest) is just an improvement on Decision Tree; Decision Tree has lot of nice properties, but it suffers from overfitting (high variance), by taking samples and ...
  • 886
4 votes
Accepted

Trade off between Bias and Variance

You want to decide this based on how well your model performs and generalizes. If your model is underfitting, you want to increase your model's complexity, increasing variance and decreasing bias. If ...
4 votes

How does C have effects on bias and variance of a Support Vector Machine?

The C being a regularized parameter, controls how much you want to punish your model for each misclassified point for a given curve. If you put large value to C it will try to reduce errors but at the ...
  • 1,401
3 votes
Accepted

Bagging vs Boosting, Bias vs Variance, Depth of trees

why we are supposed to use weak learners for boosting (high bias) whereas we have to use deep trees for bagging (very high variance) Clearly it wouldn't make sense to bag a bunch of shallow trees/...
  • 6,035
3 votes

Bias-variance tradeoff in practice (CNN)

Normally, the training loss is lower than the validation one. This does not indicate any overfitting. Indeed, it is even suspicious when you training loss is higher than the validation loss. From ...
3 votes

Whether add bias or not in a perceptron

Suppose bias as a threshold. Using threshold, your activation function moves across the $x$ axis which may get complicated. Consequently, people usually use the bias term and always centre the ...
  • 13.5k
3 votes

Linear machine learning algorithms "often" have high bias/low variance?

The "often" is the key here - the way that linear models are built, especially compared to other types of models, are more likely to favor certain types of errors.... in this case, they are more ...
  • 31
3 votes

Bagging vs pasting in ensemble learning

Let's say we have a set of 40 numbers from 1 to 40. We have to pick 4 subsets of 10 numbers. Case 1 - Bagging - We will pick the first number, put it back, and then pick the next. This makes all the ...
  • 5,224
3 votes
Accepted

Which between random forest or extra tree is best in a unbalance dataset?

Both Random Forest Classifier and Extra Trees randomly sample the features at each split point, but because Random Forest is greedy it will try to find the optimal split point at each node whereas ...
  • 354
3 votes

Why is it okay to set the bias vector up with zeros, and not the weight matrices?

As per Efficient Backprop from Lecun (§4.6) weight should be initialized in the linear region of the activation function. If they are too big, activation function will saturate and provide small ...
  • 2,204
3 votes

Difference between ethics and bias in Machine Learning

The term bias is, to my knowledge, not related to ethics in the context of ML. Instead, it usually refers either to the bias–variance tradeoff or to a learnable parameter of a model, e.g. bias in a ...
  • 5,020
2 votes
Accepted

How is the equation for the relation between prediction error, bias, and variance defined?

If: $$Err(x)=E[(Y-\hat{f}(x))^2]$$ Then, by adding and substracting $f(x)$, $$Err(x)=E[(Y-f(x)+f(x)-\hat{f}(x))^2] $$ $$= E[(Y-f(x))^2] + E[(\hat{f}(x)-f(x))^2] + 2E[(Y-f(x))(\hat{f}(x)-f(x))]$$ The ...
  • 5,714
2 votes

svm optimization problem

I think that this system of equation is incorrect. If you know that (3, -1), (3, 1) and (1, 0) are support vectors then you need to solve the next system: ...
2 votes

Why underfitting is called high bias and overfitting is called high variance?

Check out the answer provided by Brando Miranda in the following Quora question: "High variance means that your estimator (or learning algorithm) varies a lot depending on the data that you give it." ...
  • 1,181
2 votes

Model biased towards low frequency data?

With structured data, you have in general 4 challenges: (1) Missing data (2) Outliers (3) Cardinality (4) Rare values (as a rule of thumb <5%) Rare values in categorical variables tend to ...
2 votes

Bias-variance tradeoff and the uncertainty principle

No, the uncertainty principle describes a property that is specific to electrons. That electrons don't display their wave and particle properties simultaneously. Here from Wikibooks: The Heisenberg ...
2 votes

Is normalizing the validation set of time series a kind of look ahead bias?

To block the data leakage from the validation set to the training set in step (2), We should first split the data to training and validation sets, then Calculate the mean and standard deviation only ...
  • 8,667
2 votes
Accepted

Predictive modeling when output affects future input

I am afraid that such situations are fundamentally inherent in predicting/forecasting contexts; quoting from the very recent paper by Taleb et al., On single point forecasts for fat-tailed variables (...
  • 1,820
2 votes
Accepted

Neural network: does bias equal to zero, is the same as, a layer without bias?

No, they are not the same: In MLP_without_bias the bias will be zero after training, because of bias=False. In ...
  • 16.3k
2 votes

What do "Under fitting" and "Over fitting" really mean? They have never been clearly defined

You can look into the following figure to get an graphical intuition. Visit the source for detailed illustration. Source : https://www.kaggle.com/getting-started/166897
  • 743
2 votes
Accepted

Does class weighting encourage overfitting when the true class distribution is imbalanced?

Your assessment is right. You must first determine the data distribution in real-time (production) and only after that proceed with train_set, ...
  • 74
2 votes

Beginner Question on Understanding Linear Classifier

I think it is safe to state that the a $3\times 4$ matrix is used for ease of notation, in actuality the picture will for example be a $400 \times 300$ matrix of pixel values. In this case it would be ...
  • 121
1 vote

What is the defining Set in NLP

If you read the following sentence at the first line of section 6: The debiasing algorithms are defined in terms of sets of words rather than just pairs, for generality, so that we can consider ...
  • 1,166
1 vote

Learning curve using micro F-score and macro F-score

Micro calculates F score globally by counting the total true positives, false negatives and false positives. Macro calculates F score for each label and find their unweighted mean. Macro F score does ...

Only top scored, non community-wiki answers of a minimum length are eligible