11
$\begingroup$

The following question displayed in the image was asked during one of the exams recently. I am not sure if I have correctly understood the Occam's Razor principle or not. According to the distributions and decision boundaries given in the question and following the Occam's Razor the decision boundary B in both the cases should be the answer. Because as per Occam's Razor, choose the simpler classifier which does a decent job rather than the complex one.

Can someone please testify if my understanding is correct and the answer chosen is appropriate or not? Please help as I am just a beginner in machine learning

the question

$\endgroup$
  • 2
    $\begingroup$ 3.328 "If a sign is not necessary then it is meaningless. That is the meaning of Occam's Razor." From the Tractatus Logico-Philosophicus by Wittgenstein $\endgroup$ – Jorge Barrios Mar 7 at 11:25
13
$\begingroup$

Occam’s razor principle:

Having two hypotheses (here, decision boundaries) that has the same empirical risk (here, training error), a short explanation (here, a boundary with fewer parameters) tends to be more valid than a long explanation.

In your example, both A and B have zero training error, thus B (shorter explanation) is preferred.

What if training error is not the same?

If boundary A had a smaller training error than B, selecting becomes tricky. We need to quantify "explanation size" the same as "empirical risk" and combine the two in one scoring function, then proceed to compare A and B. An example would be Akaike Information Criterion (AIC) that combines empirical risk (measured with negative log-likelihood) and explanation size (measured with the number of parameters) in one score.

As a side note, AIC cannot be used for all models, there are many alternatives to AIC too.

Relation to validation set

In many practical cases, when model progresses toward more complexity (larger explanation) to reach a lower training error, AIC and the like can be replaced with a validation set (a set on which the model is not trained). We stop the progress when validation error (error of model on validation set) starts to increase. This way, we strike a balance between low training error and short explanation.

$\endgroup$
3
$\begingroup$

Occam Razor is just a synonym to Parsimony principal. (KISS, Keep it simple and stupid.) Most algos work in this principal.

In above question one has to think in designing the simple separable boundaries,

like in first picture D1 answer is B. As it define the best line separating 2 samples, as a is polynomial and may end up in over-fitting. (if I would have used SVM that line would have come)

similarly in figure 2 D2 answer is B.

$\endgroup$
2
$\begingroup$

Occam’s razor in data-fitting tasks :

  1. First try linear equation
  2. If (1) don't helps much - choose a non-linear one with less terms and/or smaller degrees of variables.

D2

B clearly wins, because it's linear boundary which nicely separates data. (What is "nicely" I can't currently define. You have to develop this feeling with experience). A boundary is highly non-linear which seems like a jittered sine wave.

D1

However I am not sure about this one. A boundary is like a circle and B is strictly linear. IMHO, for me - boundary line is neither circle segment nor a line segment,- it's parabola-like curve :

enter image description here

So I opt for a C :-)

$\endgroup$
  • $\begingroup$ I'm still unsure of why you want an in-between line for D1. Occam's Razor says to use the simple solution that works. Absent more data, B is a perfectly valid division that fits the data. If we received more data that suggests more of a curve to B's data set then I could see your argument, but requesting C goes against your point (1), since it's a linear boundary that works. $\endgroup$ – Delioth Mar 7 at 20:36
  • $\begingroup$ Because there is a lot of empty space from B line towards the left circular cluster of points. This means that any new random point arriving has a very high chance being assigned to circular cluster on the left and a very small chance for being assigned to the cluster in the right. Thus, B line is not an optimal boundary in case of new random points on plane. And you can't ignore randomness of data, because usually there is always a random displacement of points $\endgroup$ – Agnius Vasiliauskas Mar 8 at 9:39
0
$\begingroup$

I am not sure if I have correctly understood the Occam's Razor principle or not.

Let's first address Occam's razor:

Occam's razor [..] states that "simpler solutions are more likely to be correct than complex ones." - Wiki

Next, let's address your answer:

Because as per Occam's Razor, choose the simpler classifier which does a decent job rather than the complex one.

This is correct because, in machine learning, overfitting is a problem. If you choose a more complex model, you are more likely to classify the test data and not the actual behavior of your problem. This means that, when you use your complex classifier to make predictions on new data, it is more likely to be worse than the simple classifier.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.