How to maintain CBOW dataset dimension and fit it in Neural Network?

Question

I am new to neural network. I'm trying to train word embeddings without using word2vec package.

Using titles from reddit worldnews dataset I'm have done some CBOW representation.

For window size three here is some of my outputs:

0 Context : ['scores', 'killed', 'pakistan'] ---> Target: clashes
1 Context : ['japan', 'resumes', 'refuelling'] ---> Target: mission
2 Context : ['us', 'presses', 'egypt'] ---> Target: gaza
3 Context : ['presses', 'egypt', 'gaza'] ---> Target: border

For Vocab size = 513, I've collected 369 of target words against 369 of 3-grams context words. Each context word is one-hot codded of length 513.

Therefore, my dataset length becomes:
X.shape = (369, 3, 1, 513) Y.shape = (369, 1, 513)

Now I'm having trouble in fitting the data in neural network. My Neural Network model is constructed with keras.

# create model
model = Sequential()
model.add(Dense(100, input_dim=1, init=369 'uniform' , activation= 'sigmoid' ))
model.add(Dense(1, init= 'uniform' , activation= 'sigmoid' ))
model.compile(loss= 'binary_crossentropy' , optimizer= 'sgd' , metrics=['accuracy'])

#train
history = model.fit(X, Y, nb_epoch=100)

Raised Error:

ValueError: Error when checking input: expected dense_9_input to have 2 dimensions, but got array with shape (369, 3, 1, 513)

Answer 1

This shows you did not properly flatten your output before the final dense layer. Your first model.add should be an embedding layer since that represents word vectors that'll be trained. The last layer should be the dense one. Hard to tell on the spot which hidden layers to consider, but try GlobalAveragePooling and Dense with the relu activation.