I am new to neural network. I'm trying to train word embeddings without using word2vec package.

Using titles from reddit worldnews dataset I'm have done some CBOW representation.

For window size three here is some of my outputs:

0 Context : ['scores', 'killed', 'pakistan'] ---> Target: clashes
1 Context : ['japan', 'resumes', 'refuelling'] ---> Target: mission
2 Context : ['us', 'presses', 'egypt'] ---> Target: gaza
3 Context : ['presses', 'egypt', 'gaza'] ---> Target: border

For Vocab size = 513, I've collected 369 of target words against 369 of 3-grams context words. Each context word is one-hot codded of length 513.

Therefore, my dataset length becomes:
X.shape = (369, 3, 1, 513) Y.shape = (369, 1, 513)

Now I'm having trouble in fitting the data in neural network. My Neural Network model is constructed with keras.

# create model
model = Sequential()
model.add(Dense(100, input_dim=1, init=369 'uniform' , activation= 'sigmoid' ))
model.add(Dense(1, init= 'uniform' , activation= 'sigmoid' ))
model.compile(loss= 'binary_crossentropy' , optimizer= 'sgd' , metrics=['accuracy'])

history = model.fit(X, Y, nb_epoch=100)

Raised Error:

ValueError: Error when checking input: expected dense_9_input to have 2 dimensions, but got array with shape (369, 3, 1, 513)

This shows you did not properly flatten your output before the final dense layer. Your first model.add should be an embedding layer since that represents word vectors that'll be trained. The last layer should be the dense one. Hard to tell on the spot which hidden layers to consider, but try GlobalAveragePooling and Dense with the relu activation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.