Initially, I wanted to create a classification tool for some program codes in a specific environment based on the application patterns they show. I had 5 different patterns I wanted to look for and eventually my aim was to come up with some kind of statistics about this programming environment that somehow represents the trends (like %40 of the programs are for voting purposes, there has been an increase in the trading programs in the last month, etc). Now, after months of labeling data and training models, I finally obtained 5 different models each one for a different pattern that has varying scores on different metrics (I had quite an unbalanced training data for each class, so just to keep things somehow consisted, always tried to look at different metrics like f1, mcc, roc auc all at once for comparison).
Now, my question is, what do exactly my test scores tell me about my real life data when I apply my models to data that is not possible for me to label or see? Is it possible to derive different statistics as mentioned above using my trained models or am I only limited to my training & testing data? Is it at least possible to mention some kind of probability on a single program, like this is predicted to be a voting program with %X probability?
I would appreciate if you could help me figure out how to interpret the test results from a perspective of final application.