In the book "Neural Networks and Deep Learning" by Aggarwal there is an exercise 2.10.1:

Consider the following loss function for training pair $(\overline{X},y)$: $$L=max(0, a -y(\overline{W} \cdot \overline{X}))$$ The test instances are predicted as $\hat{y}=sign(\overline{W} \cdot \overline{X})$. A value of $a=0$ corresponds to the perceptron criterion and a value of $a=1$ corresponds to the SVM. Show that any value of $a>0$ leads to the SVM with an unchanged optimal solution when no regularization is used.

I am trying to solve this exercise and from the previous chapters of the book I drew a conclusion that the stochastic update for such a loss function will be $$\overline{W} \Leftarrow \overline{W} + \alpha y \overline{X} [I(y \hat{y} < a)]$$ so it's different from original SVM for cases where $y \hat{y} \in (a,1)$. For this range, the weights in original SVM will be updated, but for "modified" SVM the wegiths will not be updated. So I don't see a reason why the "modified" SVM (with $a$ instead of $1$) should be equal to the original SVM. How to solve this exercise?

  • $\begingroup$ Just a tip: you might get more and/or better answers for this kind of theoretical questions on stats.stackexchange.com $\endgroup$
    – Erwan
    Jul 29, 2019 at 14:56
  • $\begingroup$ @Erwan thanks for a tip $\endgroup$
    – DeeM
    Jul 30, 2019 at 14:19


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.