I have a multi-label classification problem where the input consist of free text, with metadata such as categories (from a fixed, limited set) associated with each text. The output consist of a set of labels. I want to try to use a decision tree for the metadata and a language model (neural network) for the text. How to combine and train the two?
Train them separately, then train a linear classifier on the probability outputs for each class, from each of the models, while keeping the models fixed?
Have them in sequence, such that the class probabilities output of one model is input features for the next? If so, which one should be upstream? And would you train the upstream one on the whole data first, then train the downstream one while keeping the first model fixed; or would you train them by alternately, i.e. training the upstream model on one batch of data, then the downstream model on the same piece of data (repeat).