0
$\begingroup$

I have an LSTM model for action recognition. During inference, any random actions that are not labelled or the model has not learned at all are also predicted with very high confidence score. I checked label smooothing technique, which will reduce the confidence score of overconfident model. Still, the score of wrongly predicted samples are quite high. How can we solve such problem?

$\endgroup$
1
  • 2
    $\begingroup$ How do you assess that the model is overconfident? $\endgroup$
    – Dave
    Commented Jan 8 at 9:54

2 Answers 2

1
$\begingroup$

If you haven't stratified the splot of train/val/test sets, I would suggest checking

  • class ratio in train set
  • class ration in val set
  • class ration in test set
$\endgroup$
0
$\begingroup$

From your description it's a little difficult to understand what you mean by "overconfident". It seems strange that your model might be very confident in predicting data that are completely out of the training distribution (ie., data "the model has not learned at all").

One potential suggestion would be to bootstrap the data to develop confidence intervals around your metrics of interest. For example, taking 10,000 random subsamples of your overall data and fitting the model for each subsample and returning the metric. See the excellent blog post here which goes over internal and external validation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.