2
Currently you're doing multiclass classification: find the most likely among N classes. Each class $C$ probability indicates how likely class $C$ is for the instance as opposed to any other class. This is why the probabilities sum to 1: in this setting, there is only one "correct" class, so two classes cannot both have high probability.
Based on ...
1
If you must select features in this way, the traditional method is to pick the number of top features that you want to obtain, instead of a threshold.
Normally this kind of feature selection is done only when there too many features with respect to the number of instances. This is why one tries to guess what would be a reasonable number of features $n$, then ...
1
Your main problem (it turns out, thanks for following up in the comments) is that you used the raw coefficients from the logistic regression as a measure of importance, but the scale of the features makes such comparisons invalid. You should either scale the features before training, or process the coefficients after.
I find it helpful to emphasize that ...
1
You get to use the entire data you have as part the training process (so the inner CV would essentially get to see all the data at some point).
The model performance estimate you get could be more stable (in the sense that it is not based on a single run using the test data, but on multiple runs.
You've covered the main benefits. However, it is important to ...
1
It is better to remove some features that have no effect or a little effect
You can use feature selection function by SKlearn (It removes all features whose variance doesn’t meet some threshold. By default, it removes all zero-variance features, i.e. features that have the same value in all samples)
Feature selection
1
So my question here is, is it wrong to point out what are the important features for classification when using relatively small dataframe with so many features?
No, it is not wrong to point out the relevant features and use a small dataframe as that is exactly the case. It will help reduce noise in your model and hence higher accuracy.
1
I think this depends in part on why you want the shap values. It can be helpful to separate (a) explaining the model, and (b) explaining the data; model explainability tools like shap only address (a), which hopefully serves as a proxy for (b).
If all you want is model explanations [(a)], then I think your approach 1 is fine; that retrained model is the one ...
1
XGBoost does not implement MAE as a loss function.
XGBoost does implement MAE as a evaluation metric.
Only top voted, non community-wiki answers of a minimum length are eligible
