I have defined the following model in Keras wherein I am trying to apply regression on some data. My input dimension is of size (300, 250) where each of the 300 values represents a cluster center and the dimension of size 250 represents the word vector embedding for the word that represents the cluster centroid. For every input sentence, I calculate the clusters that each word belongs to and thus multiply each of the above clusters by a certain weight.

        model = tf.keras.Sequential()
    input_layer = tf.keras.layers.InputLayer(batch_size=128, input_shape=(self.num_clusters, self.embedding_size))
    dense_layer = tf.keras.layers.Dense(300, activation='tanh', use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros')
    dense_layer_2 = tf.keras.layers.Dense(1, activation='tanh')
    flatten_layer = tf.keras.layers.Flatten()
    dense_layer_3 = tf.keras.layers.Dense(1, activation='tanh')

During the training process, my losses start off as being nan and continue to be nan. I am wondering if this is because I have formulated my problem incorrectly. For example, a large number of the cluster centroids (slightly more than 1/3) have a weight of 0 and thus their vector embedding ends up being 0. Could you please let me know if there is anything different that I can do in order to prevent nan losses? Thank you in advance for your help!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.