0
$\begingroup$

I am classifying documents I have around 4000 of them that I am trying to categorise into 5 categories. I am using a bag of words model which equates to about 18,000 unique words (features) and therefore I have an input layer of a neural network with 18,0000 inputs which doesn't seem right.

It is taking a huge amount of memory to try and train this network and so much time it will never converge!

Is there a way of reducing the number of input neurons seeing as a large portion of this data will be nulls?

Improve this question
$\endgroup$
1
  • $\begingroup$ One hot encoding text usually gives you a huge number of inputs. You might want to consider using word embeddings. Try word2vec or doc2vec. $\endgroup$
    – TitoOrt
    Commented Mar 19, 2018 at 15:43

1 Answer 1

Reset to default
1
$\begingroup$

Yes, actually what usually people do is to map the unique tokens in a space with fixed dimensionality, obtaining what is called "words embeddings".

And actually, using already trained word embeddings like GloVE is usually it's a best practice: those vectors are trained on huge datasets like Wikipedia or Common Crawl. A nicety of those vectors is that, thanks to the way they are built, they include also the relation between the words, a sort of semantic. This is the way I definitely suggest you to start.

Improve this answer
$\endgroup$
4
  • $\begingroup$ perfect thanks, once I have my word embeddings how would I go about adding these to my neural net? I can't find much on integrating it into a network $\endgroup$
    – Simon Nicholls
    Commented Mar 19, 2018 at 16:34
  • $\begingroup$ you use the words coordinate as features. So, if you choose a 50 dimensional space where to represent your words, you have 50 features to feed to your NN $\endgroup$
    – Vincenzo Lavorini
    Commented Mar 19, 2018 at 18:38
  • $\begingroup$ Thank you very much, I will give this a go and see what I get, I've gone down a path of convolutional neural networks as well now so will look at those too. So at the moment I have a 3 layer network, I can simply replace the input network with the 50 features (in your example)? $\endgroup$
    – Simon Nicholls
    Commented Mar 20, 2018 at 9:42
  • $\begingroup$ Yes, exactly. Simple and effective, especially if you use the pre-trained word embeddings like GloVe $\endgroup$
    – Vincenzo Lavorini
    Commented Mar 20, 2018 at 9:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.