0
$\begingroup$

I have defined the following model in Keras wherein I am trying to apply regression on some data. My input dimension is of size (300, 250) where each of the 300 values represents a cluster center and the dimension of size 250 represents the word vector embedding for the word that represents the cluster centroid. For every input sentence, I calculate the clusters that each word belongs to and thus multiply each of the above clusters by a certain weight.

        model = tf.keras.Sequential()
    input_layer = tf.keras.layers.InputLayer(batch_size=128, input_shape=(self.num_clusters, self.embedding_size))
    model.add(input_layer)
    dense_layer = tf.keras.layers.Dense(300, activation='tanh', use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros')
    model.add(dense_layer)
    dense_layer_2 = tf.keras.layers.Dense(1, activation='tanh')
    model.add(dense_layer_2)
    flatten_layer = tf.keras.layers.Flatten()
    model.add(flatten_layer)
    dense_layer_3 = tf.keras.layers.Dense(1, activation='tanh')
    model.add(dense_layer_3)

During the training process, my losses start off as being nan and continue to be nan. I am wondering if this is because I have formulated my problem incorrectly. For example, a large number of the cluster centroids (slightly more than 1/3) have a weight of 0 and thus their vector embedding ends up being 0. Could you please let me know if there is anything different that I can do in order to prevent nan losses? Thank you in advance for your help!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.