CNN to many outputs

Question

I have a dataset with 100 columns (categorial one-hot encoded) and 1 column with text data (simple sentences) and i want to build a neural network to arround 380.000 outputs labels. I have no idea what can i do.

I was thinking about a CNN with Embedding Layer and then many dense layers.

Any suggestions?

Answer 1

A shell for the type of network architecture you might be looking for could reasonably be:

{Input} -> {representation layers} -> {output}

Your question has a few implicit layers, first:

• How do I take input of various types (numeric data vs text data)
• How do I handle representations of the text data?
• How do I handle output for 380,000 output classes?

So, let's tackle these one at a time.

How do I take input of various types (numeric data vs text data)

The specific details of this are going to depend heavily on which framework you're using, as implementing this in base PyTorch is different than in Keras or Tensorflow, for example.

In any case, you're likely going to want to define different inputs for the data types you want to handle differently:

{Numeric Input}->|-- {Numeric Representation} -|
                                               |>{Dense Layer(s)} --> {Outputs}
{Text Input} --->|-- {Text Representation} ----|

How do I handle representations of the text data?

It sounds like you have an opinion for this, and that you're wanting to try:

{Numeric Input}->|-- {Numeric Representation} ->|
                                                |>{DenseLayer(s)} -> {Outputs}
{Text Input} --->|-> {CNN} -> {Embedding Layer}>|

That is a totally reasonable approach. Other approaches that form appropriate representations of text data are recurrent cells, of which one very frequently used type for text data is the LSTM.

How do I handle output for 380,000 output classes?

There's a pretty easy, straightforward way to handle 380,000 classes. You simply make the output layer a softmax (logistic) layer with 380,000 cells. Easy.

>|-- {Numeric Representation} ----->|
                                    |---> >{DenseLayer(s)} -> {Softmax, 380000}
-->|-> {CNN} --> {Embedding Layer}->|

Now, the training dataset to support an effective training of this network would need to be very, very large. Each of the 380000 classes would need to be reasonably supported by training examples in order to allow the trained network to distinguish between the classes.

Without knowing the details of the 380,000 classes, it's difficult to provide advice as to handle this difficulty (if it is one, you may have billions of training examples to work with). Common approaches are:

1. Grouping classes conceptually- for example, combining 'dog', 'cat', and 'bird' into 'Animals'

2. Limiting the classes to classes of interest. If there are relatively few classes that you care about, you can limit the class definitions to each of those and "other".