1
$\begingroup$

I was trying to train a model to predict if someone exists in a speech or not, first I extracted some set of features using a previous model, now I have a 1D vector of features for each example and I want to predict if someone exists, also the data is multilabel, so there are examples with more than one person (more the one '1' in the label vector). But the following happened:

enter image description here

It continually predict things as negative, I tried to change the model (making it deeper, shallower, change learning rate, change the activation functions) and nothing change.

So what's going on? and how to solve this issue?

I use BinaryCrossentropy loss with with_logits=True. And here is a summary of my model:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv1d (Conv1D)              (None, 3194, 256)         2048      
_________________________________________________________________
leaky_re_lu (LeakyReLU)      (None, 3194, 256)         0         
_________________________________________________________________
batch_normalization (BatchNo (None, 3194, 256)         1024      
_________________________________________________________________
conv1d_1 (Conv1D)            (None, 3190, 256)         327936    
_________________________________________________________________
leaky_re_lu_1 (LeakyReLU)    (None, 3190, 256)         0         
_________________________________________________________________
batch_normalization_1 (Batch (None, 3190, 256)         1024      
_________________________________________________________________
max_pooling1d (MaxPooling1D) (None, 638, 256)          0         
_________________________________________________________________
conv1d_2 (Conv1D)            (None, 634, 128)          163968    
_________________________________________________________________
leaky_re_lu_2 (LeakyReLU)    (None, 634, 128)          0         
_________________________________________________________________
batch_normalization_2 (Batch (None, 634, 128)          512       
_________________________________________________________________
conv1d_3 (Conv1D)            (None, 632, 64)           24640     
_________________________________________________________________
leaky_re_lu_3 (LeakyReLU)    (None, 632, 64)           0         
_________________________________________________________________
batch_normalization_3 (Batch (None, 632, 64)           256       
_________________________________________________________________
max_pooling1d_1 (MaxPooling1 (None, 158, 64)           0         
_________________________________________________________________
flatten (Flatten)            (None, 10112)             0         
_________________________________________________________________
dense (Dense)                (None, 64)                647232    
_________________________________________________________________
dense_1 (Dense)              (None, 1251)              81315     
=================================================================
Total params: 1,249,955
Trainable params: 1,248,547
Non-trainable params: 1,408

#Edit: I've tried after the answer from @Khanis to use a weighted cross entropy and things worked, but still there is too much slow learning, is that because of the huge amount of persons? if I decrease them to 100 will the problem solved?

$\endgroup$
1
  • $\begingroup$ I wonder if it has anything to do with unbalanced training data. Causing 13 less true positives, but 10000 more true negatives, seems like a good tradeoff if you only look at the absolute numbers! $\endgroup$
    – user253751
    Sep 29, 2021 at 12:01

1 Answer 1

1
$\begingroup$

Maybe you should train the model for more epochs

Or use the Weighted Binary Cross Entropy loss with a bigger weight to the Positive label

It would be easier to visualize with an input-output example, but by what you said you have a 1D vector of input features and a 1D vector of binary labels

Up to epoch 3 your model is still predicting positive labels, given by the false_positive rate > 0, it just doesn't get it right, and it is predicting less values as positives each epoch

Given the epoch log we can see that the target class is rare, on the first epoch you show

FN:2539
FP:121536
TP:131
TN:2377794

which gives

P = FN+TP = 2539+131 = 2670
N = FP+TN = 121536+2377794 = 2499330

So the positive class is rougthly 0.1% of the cases

Given that, a model that always predict Negative doesn't seem so bad

And you can see both the loss and the average error are decreasing

However, there will be a point where this can't get any better and the model may start to get things right, you just need to give it more than 10 epochs. It also depends on the quality of the previous model that extracts the features.

$\endgroup$
4
  • $\begingroup$ The features are good, I've tested the previous model. Actually, I tried to run for more epoch and the model keep predicting everything as zero, even though loss decreased to nearly zero. $\endgroup$
    – Kais Hasan
    Sep 29, 2021 at 12:39
  • $\begingroup$ And it's true that the positive is only about 0.1%, specifically there is no example with more than 3 '1', and in the average there is only 2 ones, so for each example there is about 2 ones and 1249 zeros, is that a problem? $\endgroup$
    – Kais Hasan
    Sep 29, 2021 at 12:42
  • $\begingroup$ Since the class is so rare, If you're more interested in getting some of the 1s maybe you could use the WeightedBinaryCrossEntropy as loss, and give a weight 100-1000x greater to the 1s. I've edited the answer to add that option $\endgroup$
    – Khanis Rok
    Sep 29, 2021 at 20:14
  • $\begingroup$ Thank you, the initial problem solved, but still there is a problem, can you please refer to the edit section of my question and tell me if you know something about that. $\endgroup$
    – Kais Hasan
    Sep 30, 2021 at 10:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.