My dataset is composed of an idle system that, at some time instants, receives requests. I'm trying to predict these instants through a clock. Since the requests are sparsely distributed (I've forced them to last for a while so they don't get too sparse), I wanted to create a new loss function that would penalize the model if it only gives out a zero prediction for everything. My implementation attempt is just a penalty for the standard logits:

def sparse_penalty_logits(y_true, y_pred):
    penalty = 10
    if y_true != 0:
        loss = -penalty*K.sum((y_true*K.log(y_pred) + (1 - y_true)*K.log(1 - y_pred)))
        loss = -K.sum((y_true*K.log(y_pred) + (1 - y_true)*K.log(1 - y_pred)))

    return loss

Is it correct? (I have also tried it with tensorflow). Every time I run it I either get a lot of NaN's as the loss or predictions that are not binary at all. I wonder if I'm doing something wrong at setting up the model also because binary_crossentropy is not working properly either. My model is something like this (the targets are represented by a column with either 0's or 1's):

model = Sequential()
model.add(Dense(100, activation = 'relu', input_shape = (train.shape[1],)))
model.add(Dense(100, activation = 'relu'))
model.add(Dense(100, activation = 'relu'))
model.add(Dense(1, activation = 'sigmoid'))

model.compile(optimizer = 'adam', loss = sparse_penalty_logits)

If I run it, as I said, I get very strange results (boy, do I feel like I've messed up real bad...):

Not binary at all. Boy, have I messed this up...


From the mentioned problems you are facing, this seems like a problem of exploding gradients. The exploding gradients problem can be identified by:

  • The model is unable to get traction on your training data (e.g. poor loss).
  • The model is unstable, resulting in large changes in loss from update to update.
  • The model loss goes to NaN during training.

More about Exploding gradient problem can be found at this article

I would suggest you to use some gradient clipping technique in you code and this will remove the NaN generation during model training.

| improve this answer | |
  • $\begingroup$ I've added an SGD optimizer with gradient clipping, as you suggested, with the line sgd = optimizers.SGD(lr=0.0001, clipnorm = 1, clipvalue = 0.5) (I've also tried other values for clipnorm and clipvalue). That kinda helps, but the model isn't converging consistently, nor are the predictions binary. The predictions not being binary is the weirdest thing, because I've placed a sigmoid at the end and the targets are 0's and 1's. I think I'm gonna try doing an LSTM and see what I get... $\endgroup$ – Philippe Fanaro Oct 4 '18 at 15:55
  • $\begingroup$ This seems strange. What function are you using to predict values from the model? $\endgroup$ – thanatoz Oct 4 '18 at 18:00
  • $\begingroup$ What do you mean exactly? If it's the activation function, I'm usually trying ReLU, but I've also tried tanh and sigmoid. If you mean the underlying function behind the requests, there is none, I just wanted it to basically memorize that at some time instants there would be a request coming. $\endgroup$ – Philippe Fanaro Oct 4 '18 at 20:22
  • $\begingroup$ Man, this is annoying me so much that I'm going to offer a \$15 dollars bounty on who can solve the problem. If you can do it with an LSTM I will double it to \$30. The notebook with my code can be found here. $\endgroup$ – Philippe Fanaro Oct 4 '18 at 20:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.