1
$\begingroup$

I'm trying to train a word embedding autoencoder, but it either doesn't train, or trains but doesn't make predictions. I know I'm doing something wrong, so any help is greatly appreciated.

Here is my padded sequence: (10,000, 200)

array([[ 19,  18,  15, ...,   0,   0,   0],
       [ 11,  13,  12, ...,   0,   0,   0],
       [ 11,  13,  12, ...,   0,   0,   0],
       ...,
       [ 19,  18,  15, ...,   0,   0,   0],
       [ 19,  18,   3, ...,   0,   0,   0],
       [ 48, 554,   3, ...,   0,   0,   0]])

I have the following model definition:

model = Sequential([
    InputLayer(input_shape=(200)),
    Embedding(input_dim=1500, output_dim=4, input_length=200, mask_zero=True),
    Flatten(),
    Dense(700, activation='relu'),
    Dense(500, activation='relu'),
    Dense(200, activation=???),
])

model.compile(loss='???', optimizer='nadam', metrics=['mse'])

model.fit(padded, padded, epochs=10, verbose=1, batch_size=1000)

So, I am not sure what the activation of the last layer should be. I made it "softmax" thinking that it will predict the sequence values in the padded sequence. Then, I changed the loss to "sparse_categorical_crossentropy". Though, that throws an error, because it doesn't like the "padded" dimensionality in the fit statement. I changed it by adding another dimension (np.expand_dims(padded, -1), but it still didn't like it.

Then, I changed the activation to "None", but that doesn't work, because it generates negative values - instead of the sequence values. In this case, I change the loss to 'mse'.

Neither option works. Any thoughts/suggestions are greatly appreciated. Thanks!

Note: I also tried an LSTM version of this. That is more problematic, because it quickly runs out of memory. I can share that if needed.

Edit

I did scale the problem down. So, the padded sequence is (10,000, 100). The model looks like this:

model = tf.keras.Sequential([
    tf.keras.layers.InputLayer(input_shape=(100)),
    tf.keras.layers.Embedding(input_dim=80, output_dim=4, input_length=100, mask_zero=True),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(300, activation='relu'),
    tf.keras.layers.Dense(200, activation='relu'),
    tf.keras.layers.Dense(100, activation='softmax'),  # it doesn't like 80 in this layer. It wants 100
   
])
model.compile(loss='sparse_categorical_crossentropy', optimizer='nadam', metrics=['mse'])

Now, I'm getting this error: Invalid argument: logits and labels must have the same first dimension, got logits shape [1000,100] and labels shape [100000] I tried np.expand_dims(padded, -1) but still getting the same error. I'm at a loss.

$\endgroup$
1
  • $\begingroup$ Your dataset and model might be too heavy for a start. Why don't you try on a smaller dataset and a smaller model to do an experiment and see if the basic concept is working well? $\endgroup$ Sep 17 at 13:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.