1
$\begingroup$

My data has a hierarchy structure - meaning that there is an N class at level 1 and an M class at level M. After training both models separately with a different set of data (both are Logistic regression but can be changed) I want to predict to N + M classes.

Currently, I am solving it by collecting all data before training and adding examples from level 2 to level 1 results in (N + 1) classes (+ 1 meaning all examples from level 2) and if the model at level 1 predicts that "+1" class then a send the query to the second model. But collecting the examples before training is costly and also the training set is unbalanced after collecting. Pseudocode:

clf1 = LogisticRegression()
clf1.fit(X1 + X2, Y1 + ["ID of +1 class"] * len(Y2))
clf2 = LogisticRegression()
clf2.fit(X2, Y2)
if cl1.predict(X_test_example) == ["ID of +1 class"]:
    clf2.precict(X_test_example)

But I need to preserve this hierarchical structure so the merging the two models directly is not possible. Is there a way to combine two outputs of classification models (in a form of the probability distribution) to get meaningful output are avoid problems with the overconfidence of models? Pseudocode:

clf1 = LogisticRegression()
clf1.fit(X1, Y1)
clf2 = LogisticRegression()
clf2.fit(X2, Y2)
combined_clf = clf1 + clf2
$\endgroup$
0
$\begingroup$

I can think of two options, there are probably more:

  • Train a binary classification "top model" which predicts whether the instance belongs to model 1 or model 2. The two final models only need to be trained with their respective data and only predict their respective N or M classes. The performance of the top model is crucial in this option.
  • Train an open set classification model. It's a bit similar to what you're doing currently by adding a class "other model" to the first model, except that "other" is not a regular class and you don't need to train the model with instances from this class. This is related to one-class classification and requires a specific training method which can handle open-set classification.
$\endgroup$
3
  • $\begingroup$ The first approach is not feasible because the approach should be universal and can be extended to an unknown number of levels (meaning that there can be more than two classification model which needs to be merged and I do not know how many in the time of training). $\endgroup$
    – United121
    Jan 28 at 7:12
  • $\begingroup$ The second approach is more suitable and similar to what I was also trying - DOC approach (presented in this paper arxiv.org/abs/1709.08716) which train logistic regression model with One-vs-Rest training scheme and then fit Gaussians to confidence score to get Outliers in data without need to define them in training - One-class classification seems like another approach to handle open-set classification and I will explore it more $\endgroup$
    – United121
    Jan 28 at 7:17
  • $\begingroup$ what do you think about this approach? - datascience.stackexchange.com/questions/91170/… $\endgroup$
    – United121
    Mar 26 at 10:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.