How will Occam's Razor principle work in Machine learning

Question

The following question displayed in the image was asked during one of the exams recently. I am not sure if I have correctly understood the Occam's Razor principle or not. According to the distributions and decision boundaries given in the question and following the Occam's Razor the decision boundary B in both the cases should be the answer. Because as per Occam's Razor, choose the simpler classifier which does a decent job rather than the complex one.

Can someone please testify if my understanding is correct and the answer chosen is appropriate or not? Please help as I am just a beginner in machine learning

Answer 1

Occam’s razor principle:

Having two hypotheses (here, decision boundaries) that has the same empirical risk (here, training error), a short explanation (here, a boundary with fewer parameters) tends to be more valid than a long explanation.

In your example, both A and B have zero training error, thus B (shorter explanation) is preferred.

What if training error is not the same?

If boundary A had a smaller training error than B, selecting becomes tricky. We need to quantify "explanation size" the same as "empirical risk" and combine the two in one scoring function, then proceed to compare A and B. An example would be Akaike Information Criterion (AIC) that combines empirical risk (measured with negative log-likelihood) and explanation size (measured with the number of parameters) in one score.

As a side note, AIC cannot be used for all models, there are many alternatives to AIC too.

Relation to validation set

In many practical cases, when model progresses toward more complexity (larger explanation) to reach a lower training error, AIC and the like can be replaced with a validation set (a set on which the model is not trained). We stop the progress when validation error (error of model on validation set) starts to increase. This way, we strike a balance between low training error and short explanation.

Answer 2

Occam Razor is just a synonym to Parsimony principal. (KISS, Keep it simple and stupid.) Most algos work in this principal.

In above question one has to think in designing the simple separable boundaries,

like in first picture D1 answer is B. As it define the best line separating 2 samples, as a is polynomial and may end up in over-fitting. (if I would have used SVM that line would have come)

similarly in figure 2 D2 answer is B.

Answer 3

Occam’s razor in data-fitting tasks :

1. First try linear equation
2. If (1) don't helps much - choose a non-linear one with less terms and/or smaller degrees of variables.

D2

B clearly wins, because it's linear boundary which nicely separates data. (What is "nicely" I can't currently define. You have to develop this feeling with experience). A boundary is highly non-linear which seems like a jittered sine wave.

D1

However I am not sure about this one. A boundary is like a circle and B is strictly linear. IMHO, for me - boundary line is neither circle segment nor a line segment,- it's parabola-like curve :

So I opt for a C :-)

Answer 4

I am not sure if I have correctly understood the Occam's Razor principle or not.

Let's first address Occam's razor:

Occam's razor [..] states that "simpler solutions are more likely to be correct than complex ones." - Wiki

Next, let's address your answer:

Because as per Occam's Razor, choose the simpler classifier which does a decent job rather than the complex one.

This is correct because, in machine learning, overfitting is a problem. If you choose a more complex model, you are more likely to classify the test data and not the actual behavior of your problem. This means that, when you use your complex classifier to make predictions on new data, it is more likely to be worse than the simple classifier.