2
$\begingroup$

I'm building a classification model that will need to classify into one of many possible outputs. I know in advance that I will need to add and subtract nodes from the output layer as circumstances change. Please refer me to any resource you are aware of that can help me understand potential approaches. I can retrain the model from scratch every time, of course. I'm looking for alternatives. I am only aware of two approaches:

  1. Transfer learning: Only retraining the final layer with the new desired outputs.
  2. Stacking one binary classifier for every potential output and streaming my data through all of them.

I need a level of specificity that is pretty granular, for example, I'd like to be able to identify a document that is primarily focused on a topic like "Changes in goat-herding practices in Uzbekistan." This is a made-up requirement, but the point is that it may have very specific features. Please provide input on the pros and cons of the approaches that I mentioned and add any approaches that I haven't thought of.

$\endgroup$
1
$\begingroup$

@DPCII, I don't think modifying output nodes at runtime will help you. This is because,

  • A neural network is trained on a specific dataset and to predict predefined variables only. In backpropogation ( used for optimization ), the gradients are for each weight and bias ( or any other parameters ) are calculated using the loss function ( also the objective function ).

  • For the changing the number of output nodes, you have remove some the connections coming from the previous layer which is $L_N$, suppose. But during training, the parameters of $L_{N-1}$ were optimized using the gradients of the activation functions of $L_N$.

  • Hence, removing output nodes at inference time would affect all the previous layers which in turn will affect the final output ( of the remaining nodes ).

Instead, I would suggest, to construct a neural network which predicts all the possible classes. Train this NN on various samples wherein you'll require the predictions of some desired nodes. Let me explain this by an example.

  • Suppose, we have a neural network which predicts classes $C_1, C_2,C_3,...,C_N$. For a specific sample, we only need the predictions for classes $C_1$ and $C_2$ only, given a sample $X$. For this sample, we set the label as $[ y_1 , y_2 , 0 , 0 , 0 , ... , 0].$ We only set the values for classes $C_1$ and $C_2$ and rest all the classes are set to zero.

  • Similarly, we assign labels for various samples and set only those values whose predictions are needed. Rest all values are set to zero.

  • This is actually the common classification we do for images. For a given sample of a cat, I would like to predict an array of $[1,0]$ and for a sample of dog, $[0,1]$.

Hence, we could apply this ideology instead of removing nodes and disturbing the NN.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Perhaps I used the wrong term, I was referring to transfer learning at the highest layer, which would have some similarities to the previous top layer and some differences. Also, I know that I will not be able to predict all classes ahead of time. Does that affect your response at all? $\endgroup$ – DPCII Jul 5 at 18:07
  • $\begingroup$ Neural Networks have a fixed number of nodes, connections in each layer. So, modifying the top layer ( or any other layer in that sense ) will disturb all the previous layers too. $\endgroup$ – Shubham Panchal Jul 7 at 7:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.