6
$\begingroup$

The cost function given as $\hat{\beta} = (Y - \beta X)^T (Y-\beta X)$ is used to evaluate the weights $\beta$. Here $X$ is the data and $Y$ is the output. On taking the derivative, we get the estimates of the weights. This is a Least Squares formulation.

1) Can Least Squares (LS) be used when the observation (outputs) $y_i$, $i=1,2,..,N$ number of examples are categorical? I don't quite get the picture how classification problems using LS works in terms of derivative for categorical cases.

2) Can LS be used when the data $X$ is a one-hot encoding? Would the formulation and derivative be the same?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
2
$\begingroup$

Coming to your first question:

Yes, you can but it's not advisable to use LS as a cost function for classification task since the optimization problem becomes non-convex! More on it, since your model in logistic regression would be sigmoid(a non-linear function) which means your cost function will have a lot of local optima!(think of a surface of the cost function as a Himalayan valley!), So, when you will use Gradient descent(an iterative approach) for minimizing your cost function, GD will be stuck somewhere in the local minima instead of global minimal, so you will not learn the best model parameters for your model on your data! Therefore, we change the cost function from LS to Log loss for classification task that ensures a nice bell-shaped curve surface of the cost function where the gradient descent can reach to global minima and give you the corresponding best model parameters which you can happily take it later to do predictions(on test-set)! More on it here

Much more theoretical understanding of it you could find in ISLR at page 129 4.2 Why Not Linear Regression?

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks for the answer -- it's to the point and well written. Can you please point out some references where I can understand how LS as a cost function for classification task becomes non-convex. $\endgroup$
    – Srishti M
    Commented Feb 13, 2019 at 7:53
  • $\begingroup$ here you go! 1, 2, 3 $\endgroup$
    – Anu
    Commented Feb 14, 2019 at 6:38
0
$\begingroup$

1) Answer is yes. You need (k-1) dummy variables when we deal with k categories. The logic is the same since we differentiate the cost function with respect to the parameters (beta) assuming X and Y (variables) are given.

2) Similarly, there are no differences.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I'm sorry I thought the variables $X$ are categorical. As anu's answer, logistic regression model is required. In this case, the LS cost function is not the optimal one in the sense of maximizing likelihood. $\endgroup$
    – Hohyun Jung
    Commented Feb 11, 2019 at 7:50
  • $\begingroup$ You say the "logic is the same". Same as what? Using least squares (i.e., linear regression) is not the correct approach for a classification problem even if the input variables are categorical. $\endgroup$
    – Wes
    Commented Feb 12, 2019 at 19:00
0
$\begingroup$

  1. Least squares could technically be use for categorical output, but definitely should not be. Least squares (in general, linear regression) is used for continuous output and makes several assumptions about the data that fail when using categorical output. One of the main issues is that your predicted values will probably be out of the 0-1 range, but if you are looking at a binary categorical problem, they should be within that range. Instead, you should use something like logistic regression which uses the sigmoid to bring the score within 0 and 1. Here's a visualization of this.

  2. You can use LS when one-hot-encoding categorical data as long as you have a continuous output.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.