Multiclassification with large number of labels

Question

I am attempting to build a classifier with a large input space of one hot encoded vectors. The output should be a vector of labels, with 10000 possible labels each. For example, the labels could correspond to something like: (Class 10, Class 992, Class 800, Class 23, Class 30) etc. My first attempt was to one hot encode this output as well, resulting in an output space of size (10000, 5). However, I quickly discovered that to have an output space this large, I would easily exceed the memory on my GPU (16gb).

Question: What is the best way to encode and implement such an output space in PyTorch?

My first thought was to drop the one hot encoding and encode the outputs exactly as I did above, dropping the word class obviously so that the output would be a 5 dimensional vector (10, 992, 800, 23, 30). However, I wasn't sure which loss function to use here: obviously MSE or other such regression losses would not be appropriate. After doing some research, I found someone who recommended using TensorFlow's softmax_cross_entropy_with_logits (sorry I can't seem to find where I originally saw this anymore). Using https://stackoverflow.com/questions/46218566/pytorch-equivalence-for-softmax-cross-entropy-with-logits/59197990#59197990, I was able to implement a criterion in PyTorch after removing the masked parts, that would accept tensors of size (batch_size, 5) for both labels and outputs and return a number.

However, I'm not sure I'm using this correctly. Attempting to pass the same arguments into the TensorFlow version that this is supposed to be equivalent to throws dimension errors. Also, testing this on some fabricated instances don't seem to match up with what my intuition says it should:

label = torch.tensor([[8.0], [8.0]]) # Corresponds to a two instances of class 8
out = torch.tensor([[7.0], [8.0]]) # Corresponds one incorrect prediction 
criterion(out, label) # Expecting a nonzero loss because the first prediction was incorrect
# However, returns tensor(0.)

It is not behaving as expected even with a single label, let alone five. I am thinking with five, if I can get the loss function working for one, to do a balanced sum of each individual.

Answer 1

I think your understanding of the output vector is not correct (Not just for Neural Network but any Model)

We don't encode the output to reduce the dimension. Output has not much contribution on the RAM and computation, it's the input size and dimension. Simple OHE Or just the label encoding will work(Loss function will change as per the case)

What you should decide first-
Is this a Multi-class problem or Multi-label
This will decide your Encoding i.e. [1, 1, 0, 0, 1] Or [1, 0, 0, 0, 0]
This will decide the last layer's Activation Function and the Model's Loss function

It is obvious that for such a high output dimension, you will need a very large input dataset with a good variance over all the 10K class.

To reduce output dimension -
You may use any clustering technique Or your domain knowledge to club different classes.

And then, you should plan about Torch and TensorFlow