New answers tagged feature-selection
2
370 rows is quite a few, RF does bootstrap but it is still few info.
Having too many columns will lead to a more complex model (since the algorithm will work 1 000 dimensions).
Consider doing a pipeline with all the steps and search for hyperparameters and feature selection there. https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline....
0
My first question is : for example, if the highest p value is for the X1X2 feature, is it okay to eliminate this feature even when X1 and X2 can be statistically significant ?
Of course, the interaction can have no information about the target. Per example if the problem is perfectly defined by X1 and X2. The interaction $X_1 \cdot X_2$ won't add nothing ...
2
What helps the model more, keeping all features or removing correlated ones?
There is some theory about it but in the end Machine Learning is try and error. You should give it a try with all features and then doing a feature selection to see if you are able to improve your model. What works for some models doesn´t necessarily have to work for the rest of ...
2
There is plenty of methods to calculate feature importance. I recommend trying two of them LIME and SHAP.
I don't want to copy-paste material and tutorial provided by the author so please refer to these two repositories.
1
You might want to look at conditional entropy, H(A|B) and H(B|A).
2
I think merging such correlated features and create a new one, will also be a good idea. In that way we will not lose any information.
For example, sum up the values of different correlated features and take an average of it, will be the very basic option.
3
An alternative to the one provided by @Kasra is dimensionality reduction. It's another way of solving your multicollinearity problems, while avoiding deleting variables more or less arbitrarily.
You can use simpler, linear techniques such as PCA, or more complex non-linear techniques such as Autoencoders. t-SNE is a non-linear technique that is typically ...
2
You need to remove them. Redundant features only increase the computation time, increase model complexity (with no benefit) which means making interpretation of model/analysis more sophisticated and if they are many, removing them prunes your vector space by improving the density of information in dimensions of vector space (it helps e.g. in finding nearest ...
2
In model building there is a sort of iterative workflow that you can use:
Select an appropriate model you want to build e.g. for classification maybe a XGB classifier or a logistic regression, etc.
This is important because the model by itself will determine a lot about how to wrangle your data. XGB only works with numerical features so you will have to ...
0
You can only compute chi2 between two numerical arrays. You are getting that error because you are comparing a string. Also I am not sure if it works for multiclassification also.
df = df.apply(LabelEncoder().fit_transform)
This will solve the problem for you. But there are a thousand ways to encode features and for sure other will work better for you.
0
You could use Nonlinear Least Squares, in which one of the regressors is your arctan function with two more parameters to be estimated.
In R, for example:
library(minpack.lm)
df <- datasets::airquality
my_atan <- function(x, A, B){atan((x-A)/B)}
nlsLM(Ozone ~ a + b * Temp + c * my_atan(Temp, A, B),
data = df,
start = list(a = 0, b = 0, ...
1
If the dimensions are not linearly correlated, you may use an autoencoder to perform the dimensionality reduction. Just like PCA that can perform a reconstruction, but with non-linearity. Then, you can perform classification with the latent space.
Autoencoder is a multi-dimensional auto-regressive model with a dimensional bottleneck somewhere in the middle....
0
I'm not sure if you are bound to the type of model presented in your question. However, an alternative would be to use generalised additive models (GAM), e.g. with regression splines or locale regression. These methods usually give a very good fit with non-linear patterns in $X$ and there is no need to provide parameterization of $X$ so that it is easy to ...
1
One approach would be to use an algorithm designed for non-convex problems like Bayesian optimization. However, if you have already evaluated a fine grid of parameters this is unlikely to offer significant improvement. Here is an example of how you could implement Bayesian optimization for this problem.
First, we need some data. Just for fun let’s extract ...
5
For predictive power, in general, including both shouldn't be a problem. But there is a lot of nuance here.
Foremost, if predictive power isn't all you care about: if you're making statistical inferences, or care about explainability and feature importances, then including both can cause issues. Briefly, your model may split the importance of the underlying ...
0
I'll go through your questions one by one:
is feature selection more important in KNN than in other algorithms?
I don't think it is more important for kNN than for other kinds of algorithms.
If a particular feature is not predictive in a neural network, the network will just learn to ignore it. But in KNN, it seems like it could make the prediction ...
3
I think that you need just
feature_importances = rf_gridsearch.best_estimator_.feature_importances_
This provides the feature importances for all the attribures in your dataset. For more information on this as well as other options, you may also refer to Scikit-learn official documentation.
3
Like any preprocessing step, feature selection must be carried out using the training data, i.e. the process of selecting which features to include can only depend on the instances of the training set.
Once the selection has been made, i.e. the set of features is fixed, the test data has to be formatted with the exact same features. This step is sometimes ...
1
Lasso stands for ´least absolute shrinkage and selection operator´. It has a penalty that is the absolute value and makes a lot of variables converge to cero. There is a ton of blogs that explain really well Lasso on the internet, have a look!
Elastic Net is a combination of Ridge and Lasso. So it will also reduce the variables a lot.
Ridge is a quadratic ...
0
From https://en.wikipedia.org/wiki/Shapley_value, it is possible to understand that direct computation of Shapley values is difficult with their general formula :
$$ \varphi_i(v) = \frac{1}{\text{number of players}} \sum_{\text{coalitions excluding }i} \frac{\text{marginal contribution of }i\text{ to coalition}}{\text{number of coalitions excluding } i \...
0
Check out the shap library. I think that could help you.
https://github.com/slundberg/shap
0
You can train an RNN with character embeddings. This can be done by splitting the name into sequences of chars and vectorize them numerically. If you are working with Keras, you can feed them into an Embedding() layer that will learn how to represent characters. RNN layers will then process their sequence. At the output node, your Network will perform a ...
Top 50 recent answers are included
