0
$\begingroup$

I am trying to implement an autoencoder for prediction of multiple labels using Keras. This is a snippet:

input = Input(shape=(768,))
hidden1 = Dense(512, activation='relu')(input)
compressed = Dense(256, activation='relu', activity_regularizer=l1(10e-6))(hidden1) 
hidden2 = Dense(512, activation='relu')(compressed)
output = Dense(768, activation='sigmoid')(hidden2) # sigmoid is used because output of autoencoder is a set of probabilities

model = Model(input, output)
model.compile(optimizer='adam', loss='categorical_crossentropy') # categorical_crossentropy is used because it's prediction of multiple labels
history = model.fit(x_train, x_train, epochs=100, batch_size=50, validation_split=0.2)

I ran this in Jupyter Notebook (CPU) and I am getting loss and validation loss as: loss: 193.8085 - val_loss: 439.7132
but when I ran it in Google Colab (GPU), I am getting very high loss and validation loss: loss: 28383285849773932.0000 - val_loss: 26927464965996544.0000.

What could be the reason for this behavior?

Improve this question
$\endgroup$
2
  • $\begingroup$ You say that the output is a set of probabilities and that it's a prediction of multiple labels. Are these probabilities part of a single categorical probability distribution or are they independent? $\endgroup$
    – noe
    May 12, 2021 at 6:58
  • $\begingroup$ @noe, Thank you for getting back. I didn't quite understand what you meant. What I am trying to achieve is: The train and test set contains groups of labels. When the model is given the test data, it predicts probability values. Labels in the train set that has probability value higher than threshold probability value have higher chance of occurring close to the labels in the test data. $\endgroup$
    – Animeartist
    May 12, 2021 at 7:21

1 Answer 1

Reset to default
0
$\begingroup$

You should not use the categorical cross-entropy loss, but the binary cross-entropy.

The categorical cross-entropy is meant for categorical probability distribution. In your scenario, this means that only one label would be active at the same time. If multiple labels can be active at the same time, then they are independent, and you need to compute the binary cross-entropy.

Improve this answer
$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.