How to treat sparse categorical features in a Neural Network for multiclass classification with Tensorflow?

Question

I am building a Neural Network for multiclass classification.

My dataset has 3 millions of observations. My features are 7 unordered categorical values. My problem is sparse as 4 features among the 7 can take more than 1500 different values. The label is a categorical variable that takes 2100 different values.

My model outputs a vector of 2100 probabilities through the softmax function.

I am using Tensorflow with Python and I wanted to use one-hot encoding to encode each of the features.

Do this method suffer from the sparsity of my data?

For instance, do the One-Hot encoding transform my inputs into a vector of more than 2000*4 = 8000 features? In that case, if I use a mini-batch gradient descent with a batchsize of 50, is the fact that I have a lot more features than samples (8000>>>50) a problem ("curse of dimensionality")?

Would you have a solution, and a way to implement it on Tensorflow with Python?

Answer 1

Having more features than batchsize it's not a problem indeed is the rule in many fields like computer vision. Having a very sparse dataset is also the norm in NLP. Both communities rely on dropout to prevent overfitting.

You may see an example of using drouput with Tensorflow here.