0
$\begingroup$

Problem

Suppose we have trained binary classifier and want to predict value of [x1, ..., x5] with associated timestamps [t1, ..., t5]. We get the prediction as following: [0.25, 0.99, 0.1, 0.75, 0.79].

Assume that I have the domain knowledge to say that probability of positive class must not change abruptly. Jumps like from 0.99 at t2 to 0.1 at t3 cannot occur in real application.

Questions

  1. Can I enforce smooth output constraint on (any/some) classifier?
  2. Does applying moving average on the prediction probability to smooth it make sense?
$\endgroup$
1
$\begingroup$

You can use a total variation regularizer (https://en.wikipedia.org/wiki/Total_variation_denoising), it's a penalty for abrupt changes of neighbor values. It's usually used for images, that's why its TF version (https://www.tensorflow.org/api_docs/python/tf/image/total_variation) operates with 4D tensors, but if you're writing your model in pytorch for instance, it's easy to implement that regularizer yourself. Also possibly you don't need it if you've got enough data and target values there are already smooth. Your ML algorithm would just learn that smoothness from data, the only 2 cases you'd need it is when your dataset is small or when your training targets aren't smooth, but testing targets should be smooth.

$\endgroup$
6
  • $\begingroup$ Thanks for your answer, this is really neat, but this is more related to question 2 on how to smooth the prediction , and not to how to justify smoothing of trained classifier prediction. $\endgroup$
    – ptyshevs
    Sep 13 '20 at 16:50
  • $\begingroup$ Mm, ok, I thought I answered the first question as well. Adding a regularizer to a loss function implies constraints on your solution space, that means you're really enforcing a classifier to make smooth predictions (but only if data come from the same distribution as your training samples). $\endgroup$ Sep 13 '20 at 17:08
  • $\begingroup$ So you propose to add total variation of prediction probability to loss function? $\endgroup$
    – ptyshevs
    Sep 13 '20 at 17:14
  • $\begingroup$ Yep. The total variation loss is usually used this way -- in form of a regularizer $\endgroup$ Sep 13 '20 at 17:16
  • $\begingroup$ Ok, I see. Just to clarify, I don't need to do anything with the model itself, the weights will be adjusted by optimization algorithm so as to min(loss + total_variation_penalty), correct? $\endgroup$
    – ptyshevs
    Sep 13 '20 at 17:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.