5

For predictive power, in general, including both shouldn't be a problem. But there is a lot of nuance here. Foremost, if predictive power isn't all you care about: if you're making statistical inferences, or care about explainability and feature importances, then including both can cause issues. Briefly, your model may split the importance of the underlying ...


4

_feature_importance of a random forest calculates the average feature importance across all trees in the forest. While tree.feature_importances_ is the feature importance for a single tree. Since feature importance is calculated as the contribution of a feature to maximize the split criterion (or equivalently: minimize impurity of child nodes) higher is ...


4

A random forest model is an agglomeration of Decision Trees. tree.feature_importance_ defines the feature importance for each individual tree, but model.feature_importance_ is the feature importance for the forest as a whole. The docs give the explanation for calculation as: The relative rank (i.e. depth) of a feature used as a decision node in a tree can ...


3

I think that you need just feature_importances = rf_gridsearch.best_estimator_.feature_importances_ This provides the feature importances for all the attribures in your dataset. For more information on this as well as other options, you may also refer to Scikit-learn official documentation.


3

Like any preprocessing step, feature selection must be carried out using the training data, i.e. the process of selecting which features to include can only depend on the instances of the training set. Once the selection has been made, i.e. the set of features is fixed, the test data has to be formatted with the exact same features. This step is sometimes ...


3

An alternative to the one provided by @Kasra is dimensionality reduction. It's another way of solving your multicollinearity problems, while avoiding deleting variables more or less arbitrarily. You can use simpler, linear techniques such as PCA, or more complex non-linear techniques such as Autoencoders. t-SNE is a non-linear technique that is typically ...


2

You need to remove them. Redundant features only increase the computation time, increase model complexity (with no benefit) which means making interpretation of model/analysis more sophisticated and if they are many, removing them prunes your vector space by improving the density of information in dimensions of vector space (it helps e.g. in finding nearest ...


2

In model building there is a sort of iterative workflow that you can use: Select an appropriate model you want to build e.g. for classification maybe a XGB classifier or a logistic regression, etc. This is important because the model by itself will determine a lot about how to wrangle your data. XGB only works with numerical features so you will have to ...


2

There is plenty of methods to calculate feature importance. I recommend trying two of them LIME and SHAP. I don't want to copy-paste material and tutorial provided by the author so please refer to these two repositories.


2

I think merging such correlated features and create a new one, will also be a good idea. In that way we will not lose any information. For example, sum up the values of different correlated features and take an average of it, will be the very basic option.


1

What helps the model more, keeping all features or removing correlated ones? There is some theory about it but in the end Machine Learning is try and error. You should give it a try with all features and then doing a feature selection to see if you are able to improve your model. What works for some models doesn´t necessarily have to work for the rest of ...


1

You might want to look at conditional entropy, H(A|B) and H(B|A).


1

Lasso stands for ´least absolute shrinkage and selection operator´. It has a penalty that is the absolute value and makes a lot of variables converge to cero. There is a ton of blogs that explain really well Lasso on the internet, have a look! Elastic Net is a combination of Ridge and Lasso. So it will also reduce the variables a lot. Ridge is a quadratic ...


1

One approach would be to use an algorithm designed for non-convex problems like Bayesian optimization. However, if you have already evaluated a fine grid of parameters this is unlikely to offer significant improvement. Here is an example of how you could implement Bayesian optimization for this problem. First, we need some data. Just for fun let’s extract ...


1

If the dimensions are not linearly correlated, you may use an autoencoder to perform the dimensionality reduction. Just like PCA that can perform a reconstruction, but with non-linearity. Then, you can perform classification with the latent space. Autoencoder is a multi-dimensional auto-regressive model with a dimensional bottleneck somewhere in the middle....


Only top voted, non community-wiki answers of a minimum length are eligible