# Tag Info

16

How can one understand it intuitively? Underfitting is called "Simplifying assumption" (Model is HIGHLY BIASED towards its assumption). your model will think linear hyperplane is good enough to classify your data which may not be true. consider you are shown a picture of cat 1000 times, Now you are blindfolded, No matter Whatever you are shown the 1001th ...

10

There are a lot of ways bias and variance can be minimized and despite the popular saying it isn't always a tradeoff. The two main reasons for high bias are insufficient model capacity and underfitting because the training phase wasn't complete. For example, if you have a very complex problem to solve (e.g. image recognition) and you use a model of low ...

8

What are Bias and Variance? Let's start with some basic definitions: Bias: it's the difference between average predictions and true values. Variance: it's the variability of our predictions, i.e. how spread out your model predictions are. They can be understood from this image: (source) What to do about bias and variance? If your model suffers from a bias ...

5

The tradeoff between bias and variance summarizes the "tug of war" game between fitting a model that predicts the underlying training dataset well (low bias) and producing a model that doesn't change much with the training dataset (low variance). What statisticians/mathematicians a while ago realized is that any model can be made to perfectly fit the ...

4

You want to decide this based on how well your model performs and generalizes. If your model is underfitting, you want to increase your model's complexity, increasing variance and decreasing bias. If your model is overfitting, you want to regularize the model and/or feed it more training data, decreasing variance and increasing bias.

4

Let us assume our model to be described by $y = f(x) +\epsilon$, with $E[\epsilon]=0, \sigma_{\epsilon}\neq 0$. Let furthermore $\hat{f}(x)$ be our regression function, i.e. the function whose parameters are the ones that minimise the loss (whatever this loss is). Given a new observation $x_0$, the expected error of the model is $$E[(y-\hat{f}(x))^2|x=x_0].... 4 why we are supposed to use weak learners for boosting (high bias) whereas we have to use deep trees for bagging (very high variance) Clearly it wouldn't make sense to bag a bunch of shallow trees/weak learners. The average of many bad predictions will still be pretty bad. For many problems decision stumps (a tree with a single split node) will produce ... 4 The C being a regularized parameter, controls how much you want to punish your model for each misclassified point for a given curve. If you put large value to C it will try to reduce errors but at the same time it may happen that it would not perform better on test dataset hence cause overfitting. To get to know more about effect of C in svm. Refer this. 3 Question 1: Bagging (Random Forest) is just an improvement on Decision Tree; Decision Tree has lot of nice properties, but it suffers from overfitting (high variance), by taking samples and constructing many trees we are reducing variance, with minimal effect on bias. Boosting is a different approach, we start with a simple model that has low variance and ... 3 Let's say we have a set of 40 numbers from 1 to 40. We have to pick 4 subsets of 10 numbers. Case 1 - Bagging - We will pick the first number, put it back, and then pick the next. This makes all the draw independent and consequently have very little correlation. So, if you make a Tree on the first 10 samples and another Tree on the next, both the trees will ... 3 Suppose bias as a threshold. Using threshold, your activation function moves across the x axis which may get complicated. Consequently, people usually use the bias term and always centre the activation function which is the step function at zero. There is nothing wrong in both cases. 3 Normally, the training loss is lower than the validation one. This does not indicate any overfitting. Indeed, it is even suspicious when you training loss is higher than the validation loss. From other hand, worsening of the validation accuracy while improving on the train set definitely tells you that you overfits. Generally speaking, overfitting means ... 3 The "often" is the key here - the way that linear models are built, especially compared to other types of models, are more likely to favor certain types of errors.... in this case, they are more likely to produce bias-type errors rather than variance-type. Another way of thinking about it is that the way that most linear models will give you broadly correct ... 3 Both Random Forest Classifier and Extra Trees randomly sample the features at each split point, but because Random Forest is greedy it will try to find the optimal split point at each node whereas Extra trees selects the split point randomly. I would choose Random Forest because it's more likely to create a split point that accounts for the imbalanced class, ... 2 If:$$Err(x)=E[(Y-\hat{f}(x))^2]$$Then, by adding and substracting f(x),$$Err(x)=E[(Y-f(x)+f(x)-\hat{f}(x))^2] = E[(Y-f(x))^2] + E[(\hat{f}(x)-f(x))^2] + 2E[(Y-f(x))(\hat{f}(x)-f(x))]$$The first term is the irreducible error, by definition. The second term can be expanded like this:$$E[(\hat{f}(x)-f(x))^2] = E[\hat{f}(x)^2]+E[f(x)^2] -2E[f(x)\hat{...

2

I think that this system of equation is incorrect. If you know that (3, -1), (3, 1) and (1, 0) are support vectors then you need to solve the next system: 3*w1 - 1*w2 - w0 = ±1 3*w1 + 1*w2 - w0 = ±1 1*w1 + 0*w2 - w0 = ±1 Support vectors on the same side of the separation line have the same sign on the right side of the equation (same sign here: ±1). In ...

2

With structured data, you have in general 4 challenges: (1) Missing data (2) Outliers (3) Cardinality (4) Rare values (as a rule of thumb <5%) Rare values in categorical variables tend to cause over-fitting, particularly in tree based methods. Ph.D. Data Scientist Soledad Galli has an amazing course on the subject (Udemy: "Feature Engineering". Below ...

2

No, the uncertainty principle describes a property that is specific to electrons. That electrons don't display their wave and particle properties simultaneously. Here from Wikibooks: The Heisenberg uncertainty principle states that the exact position and momentum of an electron cannot be simultaneously determined. This is because electrons simply don'...

2

To block the data leakage from the validation set to the training set in step (2), We should first split the data to training and validation sets, then Calculate the mean and standard deviation only using training set, and finally Use this mean and std to normalize both the training and validation sets. This way no information is leaked from validation ...

2

As per Efficient Backprop from Lecun (§4.6) weight should be initialized in the linear region of the activation function. If they are too big, activation function will saturate and provide small gradient step to change those weigth. If they are too small they won't really impact the gradient and make the learning too slow. Yes, if you choose the same weights ...

2

No, they are not the same: In MLP_without_bias the bias will be zero after training, because of bias=False. In MLP_with_bias_zero the bias is zero at initialization, but this will not prevent it from being updated during training.

1

There is both subjective and objective approaches to removing bias ( de-baising techniques ) from the training datasets. It is observed that the sources of bias generally arise from the following data quality issues. Over and under-sampling Skewed samples Feature choice/limited features Proxies/redundant encodings Biases and injustice in the primary data ...

1

I think you are mixing up the bias of a model as in here, with the bias terms of a neural network which are just the constant term of the linear model of each layer. Updating the biases for training will not reduce overfitting since each bias is an additional parameter of the model. Remember that the weights (and the bias is also a weight) are updated ...

1

Bias and variance are just descriptions for the two ways that a model can give subpar results. Either the model hasn't learned enough yet and its understanding of the problem is very general (bias), or it has learned the data given to it too well and cannot relate that knowledge to new data (variance). By carefully monitoring how our model is doing in ...

1

Check out the answer provided by Brando Miranda in the following Quora question: "High variance means that your estimator (or learning algorithm) varies a lot depending on the data that you give it." "Underfitting is the “opposite problem”. Underfitting usually arises because you want your algorithm to be somewhat stable, so you are trying to restrict your ...

1

A model based on simple assumptions (biased) will probably fit the data badly (under-fitting) whereas a more complex, flexible model that can vary more may fit the training data so well (over-fitting) that it becomes less good at predicting new data.

1

The bias is error of the model in the training stage. If your goal is, let's say, a 96% of accuracy, and you get 90%, you have a bias of 6%. The variance is the error difference between training and validation, so, taking the same example, if you get a validation accuracy of 80%, you have 10% variance. A linear ML algo won't work well on non-linear cases, ...

1

Because it's a costant, everything that is a costant value remains unchanged by the E, that's why you can "move" it outside. For example: If y is a costant or it is known, the E doesn't affect it, so you can "move" it outside the symbol, and write just y. The only thing that is unknown is the estimator yhat, in fact you have and not just . That's what's ...

1

Here are some things to consider. You may already be doing some of them, but it's hard to tell. As new data arrive, add them to your database. Randomly sample, with replacement, your database. All other things being equal, take a 2/3 sample and reserve 1/3 for testing. Record the training error and the prediction error. At intervals, take past training ...

Only top voted, non community-wiki answers of a minimum length are eligible