3
$\begingroup$

While learning the SVM classification I came across the regularization parameter $\lambda$: $F(w,b) = \left\lVert w\right\lVert_2^2 +\lambda \sum_{i=1}^n max(0,1-y_i(w^Tx_i +b)).$ So from what I understand, the main point of SVM is to find the hyperplane $\{w^Tx+b=0\}$separating 2 classes. How does $\lambda$ affects the hyperplane parameters w and b? I tried to plot the 2D case, but still can't get it. Can anyone give the interpretation in this case? Thanks. enter image description here

$\endgroup$
1
$\begingroup$

This is really an interesting question. In fact, many people like Svm, many use it, many understand the idea of svm " maximize the distance between the two margins "subjects to the condition of classifying all the samples correctly!! Quite easy but can get a little bit more complicated when we are dealing with none separable data and overfitting problems.

In your question, you mentioned that this parameter is helpful to perform regularization, but why we need regularization? The answer is to avoid overfitting.

But what is overfitting? Overfitting is a problem of overfitting the data. Overfitting can be more serious if you have noisy data.

In Svm, you can think about the distance between the margins as an indicator of how much you are fitting the data. If the distance is too small, then you are trying to fit " seperate between" the samples that are very close to each other. Usually, if two samples are very close to each other and they have different labels, one of them is incorrect, or it is a noisy sample.

Now, from your question, you have two parts, the first part control the distance between the margins, the second part controls the error in miss classification. Changinging " increasing" lambda will give more attention to the error rather than the distance.

The more you reduce lambda the maximum the distance between the margins you can get.

Why you do not see this clearly in your plot? Because you plot the hyper plane between the two margins, if you can plot the margins, you will see how changing lambda is affecting the distance between the margins.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.