I am building a Neural Network for multiclass classification.

My dataset has 3 millions of observations. My features are 7 unordered categorical values. My problem is sparse as 4 features among the 7 can take more than 1500 different values. The label is a categorical variable that takes 2100 different values.

My model outputs a vector of 2100 probabilities through the softmax function.

I am using Tensorflow with Python and I wanted to use one-hot encoding to encode each of the features.

Do this method suffer from the sparsity of my data?

For instance, do the One-Hot encoding transform my inputs into a vector of more than 2000*4 = 8000 features? In that case, if I use a mini-batch gradient descent with a batchsize of 50, is the fact that I have a lot more features than samples (8000>>>50) a problem ("curse of dimensionality")?

Would you have a solution, and a way to implement it on Tensorflow with Python?


Having more features than batchsize it's not a problem indeed is the rule in many fields like computer vision. Having a very sparse dataset is also the norm in NLP. Both communities rely on dropout to prevent overfitting.

You may see an example of using drouput with Tensorflow here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.