1
$\begingroup$

Im studying the perceptron algorithm. I know that we can use the weights w as the coefficients of the hyperplane that will separate the vectors that need a classifications

In every web page i read up for a detailed explanation, it's generally said that the perceptron algorithm will find the optimal weights w. But optimal in which sense ?

Improve this question
$\endgroup$
0

2 Answers 2

Reset to default
1
$\begingroup$

Media's explanation is true for regression problems. These are problems where you predict a continuous target variable.

Your image shows a classification problem. Here, the target variable takes only two values (typically -1 and 1 in the Perceptron algorithm). In that case, an optimal solution $w^*$ is a vector of weights that perfectly separates both classes. If such a solution exists, the Perceptron algorithm will find it. But: If there is one optimal solution, there are usually infinitely many other optimal solutions. You can easily see this in your image: You can move the line a little to the left or to the right, and you can rotate it a little, and it still perfectly separates the classes.

So while the Perceptron algorithm will find an optimal solution if there is one, you cannot know which one it will find. That depends on the random starting parameters.

This is different e.g. for support vector machines. Here, there is either no optimal solution or exactly one optimal solution.

Improve this answer
$\endgroup$
8
  • $\begingroup$ Take a look at here. In both regression and classification tasks, we attempt to reduce the error. Error is a function of weights, in that space we try to reduce the error. $\endgroup$
    – Green Falcon
    Commented Feb 24, 2018 at 18:51
  • $\begingroup$ Yes, you are right. But the perceptron algorithm for classification minimizes the classification error $\frac1n \sum_{i=1}^n |\hat{y}_i-y_i|.$ This is not a strictly convex function in $\hat{y}.$ Therefore the minimum is usually not unique. The Wikipedia article explains this very well and lists some sources. $\endgroup$
    – Elias Schoof
    Commented Feb 24, 2018 at 23:18
  • 1
    $\begingroup$ you are right, that is why people use cross-entropy nowadays for classification tasks. $\endgroup$
    – Green Falcon
    Commented Feb 25, 2018 at 5:59
  • $\begingroup$ Absolutely! The question is about the perceptron algorithm, however. $\endgroup$
    – Elias Schoof
    Commented Feb 25, 2018 at 9:07
  • 1
    $\begingroup$ I think we are working with different definitions of 'perceptron algorithm.' For me, the perceptron algorithm for classification is a single neuron with the signum function as the activation function and classification error as the loss function. This would be the plain vanilla version that is usually taught first to beginners in machine learning. I am aware that you can use neurons with different activation functions (sigmoid functions, etc.) and different loss functions (cross-entropy, etc.); and for those, I absolutely agree with everything you said. $\endgroup$
    – Elias Schoof
    Commented Feb 25, 2018 at 14:35
-1
$\begingroup$

Using perceptron, you specify a cost function, Mean Squared Error for regression tasks or maybe Cross Entropy for classification tasks. The input data are the constants and the weights are the parameters of your learning problem. When you specify the cost function, if you have error, the cost would be non-zero. You use algorithms like gradient descent to decrease the cost value. This is an optimization problem which you try to decrease the value of error. When we say Perceptron finds the optimal point, the reason is that the shape of cost function, e.g. MSE is convex and there is just one optimal point which gradient is zero there and the cost has the least possible value there. If you use neural networks, the cost with respect to its parameters, weights, is not convex and you usually can not find the optimal point.

I suggest you looking here and here for understanding more neural nets optimality.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.