1
$\begingroup$

My data has a hierarchy structure - meaning that there is an N class at level 1 and an M class at level M. After training both models separately with a different set of data (both are Logistic regression but can be changed) I want to predict to N + M classes.

Currently, I am solving it by collecting all data before training and adding examples from level 2 to level 1 results in (N + 1) classes (+ 1 meaning all examples from level 2) and if the model at level 1 predicts that "+1" class then a send the query to the second model. But collecting the examples before training is costly and also the training set is unbalanced after collecting. Pseudocode:

clf1 = LogisticRegression()
clf1.fit(X1 + X2, Y1 + ["ID of +1 class"] * len(Y2))
clf2 = LogisticRegression()
clf2.fit(X2, Y2)
if cl1.predict(X_test_example) == ["ID of +1 class"]:
    clf2.precict(X_test_example)

But I need to preserve this hierarchical structure so the merging the two models directly is not possible. Is there a way to combine two outputs of classification models (in a form of the probability distribution) to get meaningful output are avoid problems with the overconfidence of models? Pseudocode:

clf1 = LogisticRegression()
clf1.fit(X1, Y1)
clf2 = LogisticRegression()
clf2.fit(X2, Y2)
combined_clf = clf1 + clf2
$\endgroup$
1
  • $\begingroup$ Sounds like a variation on a regression tree where the model is a nominal one. There are variants of random forest that would do a solid job at hierarchies of nominal models. $\endgroup$ Commented Aug 12, 2023 at 1:00

2 Answers 2

0
$\begingroup$

I can think of two options, there are probably more:

  • Train a binary classification "top model" which predicts whether the instance belongs to model 1 or model 2. The two final models only need to be trained with their respective data and only predict their respective N or M classes. The performance of the top model is crucial in this option.
  • Train an open set classification model. It's a bit similar to what you're doing currently by adding a class "other model" to the first model, except that "other" is not a regular class and you don't need to train the model with instances from this class. This is related to one-class classification and requires a specific training method which can handle open-set classification.
$\endgroup$
3
  • $\begingroup$ The first approach is not feasible because the approach should be universal and can be extended to an unknown number of levels (meaning that there can be more than two classification model which needs to be merged and I do not know how many in the time of training). $\endgroup$
    – United121
    Commented Jan 28, 2021 at 7:12
  • $\begingroup$ The second approach is more suitable and similar to what I was also trying - DOC approach (presented in this paper arxiv.org/abs/1709.08716) which train logistic regression model with One-vs-Rest training scheme and then fit Gaussians to confidence score to get Outliers in data without need to define them in training - One-class classification seems like another approach to handle open-set classification and I will explore it more $\endgroup$
    – United121
    Commented Jan 28, 2021 at 7:17
  • $\begingroup$ what do you think about this approach? - datascience.stackexchange.com/questions/91170/… $\endgroup$
    – United121
    Commented Mar 26, 2021 at 10:13
0
$\begingroup$

You can use a hierarchical classification approach to solve this problem. Hierarchical classification is a method of machine learning that classifies data points into a set of predefined categories that have a hierarchical structure.

Here's how you do it:

  1. Train your level 1 model with N classes. This model will be used to predict the high-level category of a new data point.

  2. Train separate models for each of the N classes at level 1. Each of these models will be used to predict the sub-category of a new data point, given that it belongs to the corresponding high-level category.

  3. When predicting the class of a new data point, first use the level 1 model to predict the high-level category. Then, use the corresponding level 2 model to predict the sub-category.

This approach has several advantages:

  • It allows you to train each model separately, using only the data relevant to that model. This can make the training process more efficient and can also help to balance the training data for each model.

  • It allows you to make predictions at different levels of granularity, depending on your needs. For example, you might only need to predict the high-level category for some data points, but need to predict the sub-category for others.

  • It can potentially improve the accuracy of your predictions, by allowing each model to specialize in a specific part of the classification problem.

You can use any type of model for each level, depending on your needs and the nature of your data. Logistic regression could work well, but you might also consider other types of models, such as decision trees, random forests, or neural networks.

$\endgroup$
1
  • $\begingroup$ It is good idea but my situation is more dynamic - for example, I have 3 models (X, Y, Z) and only in inference time I get the information about the combination - so it can be X + Y, Y + Z or any other combination ... and I have several thousand of these models so training each combination is not feasible. $\endgroup$
    – United121
    Commented Jul 27, 2023 at 10:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.