1
$\begingroup$

I have a model I've trained on ~3400 features with ~500 samples (40:20:20 train:test:val) that I've calculated MDA on using the eli5 package. However, all the features are zero when I calculate this with the train or test dataset.

My dataset is a fairly sparse dataset with mostly zeros and I'm trying to classify disease. Here is a sample:

enter image description here

The model is a random forest that is predicting disease class (0 or 1). the dataset is imbalanced with appx (90% 0 and 10% 1)

My model accuracies are as follows:

  • train: 0.9923469387755102

  • test: 1.0

  • val: 0.9764705882352941

  • confusion matrix for test [[76 0] [ 0 8]]

these are my gini importances:

gini scores

MDAs are all zero for testing and training. Why is that and what does it mean for the generalizability of the model?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I am assuming MDA means Mean Decrease Accuracy. Generally speaking, a good performance on a validation/testing dataset means that your model generalizes well.

On the other hand MDAs of all exactly zero means either the model always needs just one variable to get its best performance no matter what variable that is (very unlikely), or that you plain simply calculated the MDAs wrong. This is impossible to tell without code/examples/etc.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.