# Word embedding autoencoder

I'm trying to train a word embedding autoencoder, but it either doesn't train, or trains but doesn't make predictions. I know I'm doing something wrong, so any help is greatly appreciated.

Here is my padded sequence: (10,000, 200)

array([[ 19,  18,  15, ...,   0,   0,   0],
[ 11,  13,  12, ...,   0,   0,   0],
[ 11,  13,  12, ...,   0,   0,   0],
...,
[ 19,  18,  15, ...,   0,   0,   0],
[ 19,  18,   3, ...,   0,   0,   0],
[ 48, 554,   3, ...,   0,   0,   0]])


I have the following model definition:

model = Sequential([
InputLayer(input_shape=(200)),
Flatten(),
Dense(700, activation='relu'),
Dense(500, activation='relu'),
Dense(200, activation=???),
])



So, I am not sure what the activation of the last layer should be. I made it "softmax" thinking that it will predict the sequence values in the padded sequence. Then, I changed the loss to "sparse_categorical_crossentropy". Though, that throws an error, because it doesn't like the "padded" dimensionality in the fit statement. I changed it by adding another dimension (np.expand_dims(padded, -1), but it still didn't like it.

Then, I changed the activation to "None", but that doesn't work, because it generates negative values - instead of the sequence values. In this case, I change the loss to 'mse'.

Neither option works. Any thoughts/suggestions are greatly appreciated. Thanks!

Note: I also tried an LSTM version of this. That is more problematic, because it quickly runs out of memory. I can share that if needed.

Edit

I did scale the problem down. So, the padded sequence is (10,000, 100). The model looks like this:

model = tf.keras.Sequential([
tf.keras.layers.InputLayer(input_shape=(100)),

Now, I'm getting this error: Invalid argument: logits and labels must have the same first dimension, got logits shape [1000,100] and labels shape [100000] I tried np.expand_dims(padded, -1) but still getting the same error. I'm at a loss.