3
$\begingroup$

I have a network to generate a single number $T$. I know in advance: a property of the loss function is that, when $T \in [a_1, a_2]$, the loss has the same value $L_1$; when $T \in [a_2, a_3]$, the loss has another value $L_2$; etc. The loss function resembles a piecewise function.

A concrete, simplified example of this problem is perhaps something like object classification. I have a set of objects, and their distances to a category $C$ that I want to classify these objects into. The distances are $[d_1, d_2, \dots, d_K]$. Assume without loss of generality that $d_1 \leq d_2 \leq \dots \leq d_K$. I want to learn a threshold $T$ for these objects that says, if the distance is near enough, then they belong to the category $C$; otherwise, they are not members of the category. For example, if $d_3 \leq T \leq d_4$, then objects $1, 2$ and $3$ (with distances $d_1, d_2, d_3$) belong to $C$.

What learning techniques I may use to learn weights of the network? Any help will be greatly appreciated.

I will next combine the above network with other differentiable learning components, so ideally it would be good if the approach is compatible with gradient descent.

$\endgroup$
2
  • 1
    $\begingroup$ Could you please provide some concrete toy example of such a problem. Otherwise I cannot understand how is this different from a usual classification task? For example, in the case of two classes, 0 and 1, the loss is 1 when the label is misclassified, and 0 when it is classified correctly. From what I know, learning is usually affected more by the form of the activation function rather than by the loss values. A toy example would be helpful in clarifying the problem. $\endgroup$ Jul 12 '20 at 8:15
  • $\begingroup$ Hey Vladislav, could you kindly take a look at the problem now? I have added an example. $\endgroup$ Jul 15 '20 at 7:21
0
$\begingroup$

I would consider Cubic splines to approximate your piecewise function.
https://en.wikiversity.org/wiki/Cubic_Spline_Interpolation#:~:text=Cubic%20spline%20interpolation%20is%20a,Lagrange%20polynomial%20and%20Newton%20polynomial.
You can regularize your spline by adding continuity constraint on the first and the second derivatives at the nodes so the function would be smooth and would be a good approximation to your piecewise loss.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.