I'm trying to train a word embedding autoencoder, but it either doesn't train, or trains but doesn't make predictions. I know I'm doing something wrong, so any help is greatly appreciated.
Here is my padded sequence: (10,000, 200)
array([[ 19, 18, 15, ..., 0, 0, 0], [ 11, 13, 12, ..., 0, 0, 0], [ 11, 13, 12, ..., 0, 0, 0], ..., [ 19, 18, 15, ..., 0, 0, 0], [ 19, 18, 3, ..., 0, 0, 0], [ 48, 554, 3, ..., 0, 0, 0]])
I have the following model definition:
model = Sequential([ InputLayer(input_shape=(200)), Embedding(input_dim=1500, output_dim=4, input_length=200, mask_zero=True), Flatten(), Dense(700, activation='relu'), Dense(500, activation='relu'), Dense(200, activation=???), ]) model.compile(loss='???', optimizer='nadam', metrics=['mse']) model.fit(padded, padded, epochs=10, verbose=1, batch_size=1000)
So, I am not sure what the activation of the last layer should be. I made it "softmax" thinking that it will predict the sequence values in the padded sequence. Then, I changed the loss to "sparse_categorical_crossentropy". Though, that throws an error, because it doesn't like the "padded" dimensionality in the fit statement. I changed it by adding another dimension (np.expand_dims(padded, -1), but it still didn't like it.
Then, I changed the activation to "None", but that doesn't work, because it generates negative values - instead of the sequence values. In this case, I change the loss to 'mse'.
Neither option works. Any thoughts/suggestions are greatly appreciated. Thanks!
Note: I also tried an LSTM version of this. That is more problematic, because it quickly runs out of memory. I can share that if needed.
I did scale the problem down. So, the padded sequence is (10,000, 100). The model looks like this:
model = tf.keras.Sequential([ tf.keras.layers.InputLayer(input_shape=(100)), tf.keras.layers.Embedding(input_dim=80, output_dim=4, input_length=100, mask_zero=True), tf.keras.layers.Flatten(), tf.keras.layers.Dense(300, activation='relu'), tf.keras.layers.Dense(200, activation='relu'), tf.keras.layers.Dense(100, activation='softmax'), # it doesn't like 80 in this layer. It wants 100 ]) model.compile(loss='sparse_categorical_crossentropy', optimizer='nadam', metrics=['mse'])
Now, I'm getting this error:
Invalid argument: logits and labels must have the same first dimension, got logits shape [1000,100] and labels shape 
np.expand_dims(padded, -1) but still getting the same error. I'm at a loss.