# Decision boundary in a classification task

I have 1000 data points from the bivariate normal distribution $$\mathcal{N}$$ with mean $$(0,0)$$ and variance $$\sigma_1^2=\sigma_2^2=10$$ with the covariances being $$0$$. Also there are 20 more points from another bivariate normal distibution with mean $$(15,15)$$ with variance $$\sigma_1^2=\sigma_2^2=1$$ and with the covariances being $$0$$ again. I used the least squares method to calculate the parameters of the decision bounday $$\theta_0 + \theta_1 x_1 + \theta_2 x_2=0$$, that is $$\theta = (X^T X)^{-1}(X^Ty)$$ where $$y$$ is a column matrix with labels $$+1$$ for points from the first class and $$-1$$ for points from the second. The resuliting plot is as follows: It is obvious that the decision boundary failed to be correct, as it passes right through class $$-1$$ and therefore it won't classify correctly future points that might stem from the same distribution. Now, there is the issue of why this is happenning. I understand that the main problem here is the imbalance of the data set, as there are $$1000$$ points from one class but only $$20$$ from the other. This, intuitively, makes sense.

What I want someone to help me with, if possible, is to understand how this imbalance problem is incorporated into the process of minimizing the least squares cost function $$J(\theta)=\sum_{n=1}^{200}(y_n-\theta^T x_n)^2$$

How does the fact that there are only $$20$$ points from the second class causes the minimzation task $$\frac{\partial J(\theta)}{\partial \theta}=0$$ to "fail"? How do the insufficient amount of these points causes this line to pass right through them? If there is some mathematical way to show me this, it would be nice, as I already got the intuition.

• Thank you for the reply. However I am not asking for alternatives. Im asking how, in this particular example, the imbalance in the data set is causing the decision boundary to appear where it appears. And Im looking for a $\textit{mathematical}$ way to show this. Thank you Oct 21 '20 at 22:12