I have defined the following model in Keras wherein I am trying to apply regression on some data. My input dimension is of size (300, 250) where each of the 300 values represents a cluster center and the dimension of size 250 represents the word vector embedding for the word that represents the cluster centroid. For every input sentence, I calculate the clusters that each word belongs to and thus multiply each of the above clusters by a certain weight.
model = tf.keras.Sequential() input_layer = tf.keras.layers.InputLayer(batch_size=128, input_shape=(self.num_clusters, self.embedding_size)) model.add(input_layer) dense_layer = tf.keras.layers.Dense(300, activation='tanh', use_bias=True, kernel_initializer='glorot_uniform', bias_initializer='zeros') model.add(dense_layer) dense_layer_2 = tf.keras.layers.Dense(1, activation='tanh') model.add(dense_layer_2) flatten_layer = tf.keras.layers.Flatten() model.add(flatten_layer) dense_layer_3 = tf.keras.layers.Dense(1, activation='tanh') model.add(dense_layer_3)
During the training process, my losses start off as being nan and continue to be nan. I am wondering if this is because I have formulated my problem incorrectly. For example, a large number of the cluster centroids (slightly more than 1/3) have a weight of 0 and thus their vector embedding ends up being 0. Could you please let me know if there is anything different that I can do in order to prevent nan losses? Thank you in advance for your help!