I am very confused between one hot encoding and word embedding in terms of structure of the network and how it reduces the dimensionality.
I am currently using encog with c# which has some documentation but not many differing examples.
If I take an example of text classification by neural networks and have: 1000 different texts 10,000 unique words
What I was previously doing was taking each text and then working out the unique words and putting them in a list. I would then work out the number of times each unique word appeared in a text and I would end up with a matrix of:
1000 x 10,000 i.e. for each text a count of the number of times each unique word appears (text as rows and unique words (features) as columns).
I would then normalise this and use equilateral encoding (similar to one hot).
My network would then consist of 10,000 input neurons (one for each word (feature)) and then a hidden layer and then an output layer trying to predict a classification of an output column with 5 different values.
This appears to be totally wrong but it did train and pretty accurately. I'm trying to work out with one hot encoding what exactly that would mean in terms of matrix sizes, what those matrices consist of and the layout of the network.
This is also related to using word embedding as when I use this I get the following size: 1000 (number of texts), 872 (text size), 300 (vector size). I therefore have 3 dimensions so that made me think in one hot encoding I should probably have: 1000, 872, 10000 (unique words).
If someone could explain where I am going wrong here and the difference between how I would approach one hot encoded text classification vs an embedded layer using:
1000 texts, 10000 unique words, 872 text length and 300 vector size
