I am currently working on a Machine-Learning Model. In order to explain how it works, I have looked at Partial Dependence Plots, Feature Importance and all kinds of methods, but one thing still concerns me: Should I evaluate these on the dataset I trained the model with, or the current dataset on which it performs predictions? This of course leads to different results depending on the distribution and the performance of the model. How do you handle this? Any opinions or input would be highly appreciated!
1
-
$\begingroup$ The following Kaggle kernel should be helpful to you: kaggle.com/mjbahmani/santander-ml-explainability $\endgroup$– SidCommented Jul 21, 2019 at 6:52
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
You can look at the different evaluations as evaluation of the different stages of the model.
- by evaluating the model's performance on the training dataset, you could assess how and what the model has learned from the data structure. This evaluation is mainly relevant for your research stage and the reporting of your results.
- by evaluating the model's performance on the current/test dataset, you would see how the model is behaving and performing on current data the real predictions. This may help you to asses that
These two may differ in cases where the current data is behaving differently than the data that the model been trained on, and to help you to understand that the model needs to be updated and retrained on the data that it is performing poorly on. If you're not monitoring your current data feed (distribution, range of values, the ratio of missings), I would suggest you do both for assessing if your ongoing observed performance is not what you are expecting.
$\begingroup$
$\endgroup$
There is plethora of content available now. I have summarised them all. you will be amazed by number of algorithms and frameworks already available-
