New answers tagged decision-trees
1
Here is a good explanation of Gini impurity: link. I don't see why it can't be generalized to multinary splits.
The binary split is the easiest thing to do (e.g. discussion: link). That's why it is implemented in mainstream frameworks and described in countless blog posts.
A non-binary split is equivalent to a sequence of binary splits (e.g. link). However, ...
0
Advantages:-
1.) Faster than RF.
2.) Relatively easy to interpret than other algo's, although most of the
algorithms are interpretable more or less.
3.) Easy to visualize.
4.) Can have control over feature selection if you don't want to go over the
filter based or wrapper based feature selection.
5.) Easy to implement.
6.) Easy to tune the hyperparameters.
7....
1
To answer your question,
$S$ in shannon entropy represents a discrete random variable with values $s_{1},s_{2},..s_{n}$
$S$ in Information Gain represents set of training examples, in the form ${\displaystyle ({\textbf {s}},t)=(s_{1},s_{2},s_{3},...,s_{k},t)}$, where ${\displaystyle s_{a}\in vals(a)}$ is the value of the ${\displaystyle a^{\text{th}}}$ ...
1
From the graph It is very clear that a non-linear model will perform well to distinguish between Class A and Class B . A Linear model ( Logistic Regression) give an accuracy around 50% for such datasets . A non-linear model (For example -SVM ) with a kernel trick can give you a very good accuracy . Follow this link to see the practical difference between ...
0
In order to fix overfitting, you can try the following things:
1.) Cross validation
2.) Get more data (won't work everytime)
3.) Remove redundant features
4.) Early stopping rounds if you are using GBM or DL
5.) Regularization (for example Ridge or Lasso in the case of Linear Regression)
6.) Perform extensive Feature engineering
0
All GridSearch does is it looks for the best performing model among the parameters you have supplied it with. It won't fix overfitting for you.
Overfitting happens when the model is to well adjusted to the training data.
In case of SVM the model with C=1000 would definitely overfit and that is why it was not the best one. C=0.1 would probably underfit and ...
0
One way to compare models is to look at the different decision boundaries the different models have learned. The different decision boundaries can impact the evaluation metrics.
0
A few thoughts:
The first thing I would check is whether the other models overfit. You could check this by comparing the performance between the training set and the test set.
Also there's something a bit strange about k-NN always predicting the majority class. This would happen only if any instance is always closer to more majority instances than minority ...
2
There is no free lunch among Catboost, XGBoost and LightGBM. In my experience, some cases I found that XGBoost outperform other, some cases for LightGBM, and the rest for CatBoost. So there is no exact the best model until you test them all in your dataset with doing hyper parameters tune to your model.
The only think clearly better from both CatBoost and ...
0
As mentioned in the comments, check the training and test/validation metrics and compare them. If the training metric is much better than the test/validation metric, then there are high chances you are overfitting.
Another way to check for overfitting is to plot learning curves. They are basically curves for model performance. Check out the sklearn page for ...
1
You can indeed use other weak learners (as the components of an ensemble are commonly called) than just decision trees. That said, decision tree ensembles are most widely used, especially gradient boosted trees and random forest.
Sometimes, other ensembles are just a conceptual tool to facilitate analysis of algorithms, like when you're trying to understand ...
6
all the ensemble models I came through so far use/described using the decision tree.
Random Forest is the "ensemble version" of decision trees. It's a commonly used ensemble method because it's built in the algorithm itself. However ensemble methods are much more general than decision trees and can be used with any learning method, for example by ...
2
At the highest level of abstraction, the answer is yes.
You can send a set of values to be scored to every model in an ensemble, and then combine the resulting of set of scores into a single score according to a predetermined formula.
Formally speaking, every transaction follows the same path through the system, so no decision is involved.
1
If this is for classification task you can try Chi-squared test.
https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.chi2.html
Top 50 recent answers are included