What are Bias and Variance?
Let's start with some basic definitions:
Bias: it's the difference between average predictions and true values.
Variance: it's the variability of our predictions, i.e. how spread out your model predictions are.
They can be understood from this image:
What to do about bias and variance?
If your model suffers from a bias ...
The C being a regularized parameter, controls how much you want to punish your model for each misclassified point for a given curve.
If you put large value to C it will try to reduce errors but at the same time it may happen that it would not perform better on test dataset hence cause overfitting.
To get to know more about effect of C in svm. Refer this.
Both Random Forest Classifier and Extra Trees randomly sample the features at each split point, but because Random Forest is greedy it will try to find the optimal split point at each node whereas Extra trees selects the split point randomly.
I would choose Random Forest because it's more likely to create a split point that accounts for the imbalanced class, ...
As per Efficient Backprop from Lecun (§4.6) weight should be initialized in the linear region of the activation function. If they are too big, activation function will saturate and provide small gradient step to change those weigth. If they are too small they won't really impact the gradient and make the learning too slow.
Yes, if you choose the same weights ...
No, they are not the same:
In MLP_without_bias the bias will be zero after training, because of bias=False.
In MLP_with_bias_zero the bias is zero at initialization, but this will not prevent it from being updated during training.
Okay so let's start with the first question:
Is that mean bias(b) is the distance between some particular point of the red line(as per picture) to the true point(say a blue or green point).
You'd be correct had you used the word difference rather than distance. Bias is the difference between the estimated value and the true value. Think of it in this way, ...
The decision trees used in gradient boosting are typically shallow decision trees (with only a few nodes). Limiting the depth or number of nodes in the decision tree makes them simple. This is different from fully developed decision trees used as standalone models.
I am afraid that such situations are fundamentally inherent in predicting/forecasting contexts; quoting from the very recent paper by Taleb et al., On single point forecasts for fat-tailed variables (open access, para 3.7):
3.7. Forecasts can result in adjustments that make forecasts less accurate
It is obvious that if forecasts lead to adjustments, and ...
All the parameters are updated after a batch, there is no notion of order of update.
The batch can or cannot be in the same order. It could lead to overfitting in the sense of the network learn the order of the dataset. An easy turnaround is to shuffle the batch between each epochs.
No - non-parametric methods only means that the method does not assume a function form of the data. There are non-parametric methods such as Random Forest that do not always overfit. In fact nonparametric methods could underfit, it could lack the ability to fit the training data. An example of this would be a decision stump.
There is both subjective and objective approaches to removing bias ( de-baising techniques ) from the training datasets. It is observed that the sources of bias generally arise from the following data quality issues.
Over and under-sampling
Feature choice/limited features
Biases and injustice in the primary data ...