1
$\begingroup$

Initially, I wanted to create a classification tool for some program codes in a specific environment based on the application patterns they show. I had 5 different patterns I wanted to look for and eventually my aim was to come up with some kind of statistics about this programming environment that somehow represents the trends (like %40 of the programs are for voting purposes, there has been an increase in the trading programs in the last month, etc). Now, after months of labeling data and training models, I finally obtained 5 different models each one for a different pattern that has varying scores on different metrics (I had quite an unbalanced training data for each class, so just to keep things somehow consisted, always tried to look at different metrics like f1, mcc, roc auc all at once for comparison).

Now, my question is, what do exactly my test scores tell me about my real life data when I apply my models to data that is not possible for me to label or see? Is it possible to derive different statistics as mentioned above using my trained models or am I only limited to my training & testing data? Is it at least possible to mention some kind of probability on a single program, like this is predicted to be a voting program with %X probability?

I would appreciate if you could help me figure out how to interpret the test results from a perspective of final application.

$\endgroup$
2
$\begingroup$

Evaluation is a crucial part of any serious ML project, but when it comes to evaluation choices there is no perfect answer. Generally speaking one evaluates a system in order to know how well it performs (i.e. how reliable are its predictions), usually to know which level of quality to expect when used in production (but not only). However the evaluation results are only useful to some extent:

  • the evaluation method/measures should be chosen so that it actually represents "quality" for the target task. In general this is imperfect because no evaluation score can fully represent the diversity of a specific task (and that's assuming good evaluation choices)
  • the test data should be representative of the production data, and in reality it rarely is from the same distribution.
  • There's almost always a chance factor that one tries to eliminate using cross-validation or other methods, but again that's imperfect.

Besides these unavoidable simplifications, an evaluation score can be interpreted for exactly what it is. For example a precision score of 80% for class X means that one should expect an instance predicted X as truly of class X in 80% of the cases in average. Predicting a specific probability for a particular prediction requires either using a probabilistic model or devising a specific model which predicts a confidence score/probability... But in both cases the model could still can be wrong about the probability it predicts!

$\endgroup$
2
  • $\begingroup$ That was very nicely explained, thank you very much. Regarding your precision example, is it then possible to give some kind of +/- range about the probability if 80% is the average? $\endgroup$ Apr 1 '20 at 7:32
  • $\begingroup$ Yes, a margin of error can be estimated if cross-validation (or similar) is used: based on CV results the mean and standard dev. of the performance can be calculated. Afaik the distribution of the performance is usually a normal distribution, so from there it becomes possible to estimate confidence intervals etc. $\endgroup$
    – Erwan
    Apr 1 '20 at 9:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.