Least Squares optimization

The cost function given as $$\hat{\beta} = (Y - \beta X)^T (Y-\beta X)$$ is used to evaluate the weights $$\beta$$. Here $$X$$ is the data and $$Y$$ is the output. On taking the derivative, we get the estimates of the weights. This is a Least Squares formulation.

1) Can Least Squares (LS) be used when the observation (outputs) $$y_i$$, $$i=1,2,..,N$$ number of examples are categorical? I don't quite get the picture how classification problems using LS works in terms of derivative for categorical cases.

2) Can LS be used when the data $$X$$ is a one-hot encoding? Would the formulation and derivative be the same?

Coming to your first question:

Much more theoretical understanding of it you could find in ISLR at page 129 4.2 Why Not Linear Regression?

• Thanks for the answer -- it's to the point and well written. Can you please point out some references where I can understand how LS as a cost function for classification task becomes non-convex. – Srishti M Feb 13 at 7:53
• here you go! 1, 2, 3 – anu Feb 14 at 6:38

1) Answer is yes. You need (k-1) dummy variables when we deal with k categories. The logic is the same since we differentiate the cost function with respect to the parameters (beta) assuming X and Y (variables) are given.

2) Similarly, there are no differences.

• I'm sorry I thought the variables $X$ are categorical. As anu's answer, logistic regression model is required. In this case, the LS cost function is not the optimal one in the sense of maximizing likelihood. – Hohyun Jung Feb 11 at 7:50
• You say the "logic is the same". Same as what? Using least squares (i.e., linear regression) is not the correct approach for a classification problem even if the input variables are categorical. – Wes Feb 12 at 19:00
1. Least squares could technically be use for categorical output, but definitely should not be. Least squares (in general, linear regression) is used for continuous output and makes several assumptions about the data that fail when using categorical output. One of the main issues is that your predicted values will probably be out of the 0-1 range, but if you are looking at a binary categorical problem, they should be within that range. Instead, you should use something like logistic regression which uses the sigmoid to bring the score within 0 and 1. Here's a visualization of this.

2. You can use LS when one-hot-encoding categorical data as long as you have a continuous output.