0
$\begingroup$

I have 4 class binary classification models. That models identify which class a particular students is suitable for.

For example, we have user 1 and 4 classes recommendation model.

Models were identify how this user would like to take its class.

By reading user 1's personal profile data (features), model A, B, C, D predicts each class' fitness. Binary classification threshold were all 50%.

  • model A: 77%, True
  • model B: 65%, True
  • model C: 33%, False
  • model D: 88%, True

Based on this result, system recommends class A, B, and C to user 1.

However, models' performance were all different. Each model may have different F1-score, for example, model A: 77%, model B: 64%, model C: 81%, and model D: 55%.

How can we measure each recommendation score rationally, based on models' F1 score?

I also had thought that some recommender system might works, however recommendation algorithms were limit to utilize user's profile.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
1
$\begingroup$

Why compare them at all?

The four models are tackling different problems. So it seems like you should not compare these models against themselves.

If you had, for example, 3 different models to classify class A, then you could compare these models since they are all trying to solve the same problem.

It is difficult to compare the models A, B, C and D because they have different training samples and inherently the cases may be more or less difficult. For example, suppose class A is "English for Beginners" and class B is "Advanced Data Science", then maybe the student's nationality maybe good for predicting class A, but the student's past courses are needed for a good prediction of B. So, maybe scoring >90% on A is easy because you have all the info, but scoring >90% on B is extremely difficult. In the end, if you have a model_A with score 87.32% and model_B with score 84.15%, who is to say that model_A is better? We cannot because the problems they solve are different.

On the other hand, if you have model_A1, model_A2, and model_A3 that solve the same problem, you can compare them against the same test set.

Single problem

Another way to look at your problem, is as a single problem.

While you have small models like model_A, model_B, model_C and model_D, they could be combined into a single model.

You can take the results of each of the models and create a final output vector (e.g. [1,1,0,1]) which means to recommend A, B and D.

You then use this result to give a score to your whole system.

You can still use the individual scores to see where to fine-tune, but perhaps reporting a score for the whole system is desired.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thank you for sharing @Bruno. I agree with your point of view. Some of your point is right, however, without using different class label, all profile of students (X_features) were same. And would you give me an some of ml algorithms for Single problem you suggested? That ideas looking good output looks like many of choices classification problem. $\endgroup$
    – Sogo
    Jun 4, 2021 at 2:41
  • $\begingroup$ To make every thing into a single problem you have two options. Option 1) As you found out, you can use a single multi label model. Option 2) Keep the individual models (maybe model_A is a decision tree, and model_B is a neural network) and pass your input into all models, then combine the output of all the models in to a vector. Then you can report the performance of the individual models and the performance of the all the models as a single system. $\endgroup$
    – Bruno Lubascher
    Jun 7, 2021 at 6:09
0
$\begingroup$

I founds some information about this problem. Like this problem called, Multilabel Classification. Unlike multiclass classification, multilabel classification classifies labels including both 1s' or all zeros.

You can refer this ideas from sklearn: https://scikit-learn.org/stable/modules/multiclass.html

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.