1
$\begingroup$

I have 1000 data points from the bivariate normal distribution $\mathcal{N}$ with mean $(0,0)$ and variance $\sigma_1^2=\sigma_2^2=10$ with the covariances being $0$. Also there are 20 more points from another bivariate normal distibution with mean $(15,15)$ with variance $\sigma_1^2=\sigma_2^2=1$ and with the covariances being $0$ again. I used the least squares method to calculate the parameters of the decision bounday $\theta_0 + \theta_1 x_1 + \theta_2 x_2=0$, that is $$\theta = (X^T X)^{-1}(X^Ty)$$ where $y$ is a column matrix with labels $+1$ for points from the first class and $-1$ for points from the second. The resuliting plot is as follows:

enter image description here

It is obvious that the decision boundary failed to be correct, as it passes right through class $-1$ and therefore it won't classify correctly future points that might stem from the same distribution. Now, there is the issue of why this is happenning. I understand that the main problem here is the imbalance of the data set, as there are $1000$ points from one class but only $20$ from the other. This, intuitively, makes sense.

What I want someone to help me with, if possible, is to understand how this imbalance problem is incorporated into the process of minimizing the least squares cost function $$J(\theta)=\sum_{n=1}^{200}(y_n-\theta^T x_n)^2$$

How does the fact that there are only $20$ points from the second class causes the minimzation task $\frac{\partial J(\theta)}{\partial \theta}=0$ to "fail"? How do the insufficient amount of these points causes this line to pass right through them? If there is some mathematical way to show me this, it would be nice, as I already got the intuition.

$\endgroup$
0
$\begingroup$

For those data points, a threshold just on x1 axis would perfectly separate the two distributions. You could fit a decision stump to calculate the single parameter of the decision boundary.

$\endgroup$
1
  • $\begingroup$ Thank you for the reply. However I am not asking for alternatives. Im asking how, in this particular example, the imbalance in the data set is causing the decision boundary to appear where it appears. And Im looking for a $\textit{mathematical}$ way to show this. Thank you $\endgroup$
    – thenac
    Oct 21 '20 at 22:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.