2
$\begingroup$

I have heard that each of the different classification algorithms can be expressed as a neural network architecture. How can the different algorithms like Logistic Regression, SVM(Support Vector Machine), ELM(Extreme Learning Machine) be expressed as a neural network? Is there a way to convert the vector equation of a classification algorithm into neural network architecture?

$\endgroup$
2
$\begingroup$

I have heard that each of the different classification algorithms can be expressed as a neural network architecture.

The Universal Approximation Theorem guarantees that neural networks can approximate any function on a closed subset of $\mathbb{R}^n$. So we are theoretically guaranteed that any function specified by a logistic regression model, SVM, ELM, or any other model can also be expressed by a neural network.

Is there a way to convert the vector equation of a classification algorithm into neural network architecture?

This part may be hard depending on the model. Although the universal approximation theorem guarantees that an equivalent network exists, it offers no information about the network's weights or number of hidden units. Nonetheless, let's give it a shot for the three models you mentioned:

ELMs

This one is easy. ELMs are already neural networks, so there's nothing left to be done.

Logistic Regression

This one is also pretty straightforward. For simplicity, let's consider a binary classification problem. A logistic regression model can be written as $$ p(Y=1) = \frac{1}{1 + e^{-(w_0 + w_1x_1 + . . . w_nx_n)}} $$

Where the expression $p(Y=1)$ means "the probability that the class is positive". The input features are ($x_1$, $. . .$ $x_n$) and they have corresponding weights ($w_1$, . . . $w_n$).

We can represent this function with just a single neuron! Suppose we have a neuron with $n$ inputs, weights ($w_1$, . . . $w_n$), and bias term $w_0$. The output of the neuron (with no activation function) is given by $$ z = w_0 + \sum_{i=1}^{n} w_ix_i $$

Applying a sigmoid activation function to $z$ results in the exact same expression as we had above for the logistic regression model: $$ S(z) = \frac{1}{1+e^{-z}} = \frac{1}{1 + e^{-(w_0 + w_1x_1 + . . . w_nx_n)}} = P(Y=1) $$

Here's a diagram showing the neuron:

The situation would be more complex for multinomial logistic regression, but you get the idea.

SVMs

This one is probably the trickiest of the three, since an SVM's kernel function can be non-linear. Again, let's consider a binary classification problem.

A linear SVM is specified by the maximal-margin hyperplane between the two classes, which can be written as $\vec{w} \cdot \vec{x} - w_0 = 0$, where $\vec{w} = \langle w_1, . . . w_n \rangle$ and $\vec{x} = \langle x_1, . . . x_n \rangle$.

Of course, the actual classification is determined by which side of the hyperplane a point lies, so the classifier can be written $f(\vec{x}) = sign(\vec{w} \cdot \vec{x} - w_0)$, with 1 denoting the positive class and -1 denoting the negative class.

As with logistic regression, it's pretty easy to represent the decision boundary with a single neuron. A neuron with inputs $\vec{x} = \langle x_1, . . . x_n \rangle$ with weights $\vec{w} = \langle w_1, . . . w_n \rangle$ and with bias term $\hat{w_0} = -w_0$ describes the decision boundary of a linear SVM. This is the same neuron as the one associated with logistic regression, except there is no activation function and the bias term has its sign flipped:

However, it's exceedingly common for SVMs to use the kernel trick to find non-linear decision boundaries. When the kernel is non-linear, the translation of an SVM to a neural network is not so straightforward. I imagine you could find an equivalent network by using the kernel function as the activation function for each neuron, but off-hand I can't describe exactly how it would work.

Hopefully that helps!

$\endgroup$
  • $\begingroup$ can you also draw the architecture in your answer for the algorithms $\endgroup$ – codeczar Nov 7 at 19:33
  • 1
    $\begingroup$ I added diagrams for logistic regression and SVM with linear kernel. For an ELM, the architecture can vary. To draw the architecture, we would need a specific example of an ELM. $\endgroup$ – zachdj Nov 7 at 20:01
  • $\begingroup$ thanks can you also explain soft-margin SVM architecture $\endgroup$ – codeczar Nov 7 at 20:15
  • $\begingroup$ In the context of translating a trained SVM to a neural network, there is no difference between soft-margin and hard-margin SVM. At the end of the day, your SVM will have a decision boundary which is defined by a hyperplane $\vec{w} \cdot \vec{x} - w_0$. That hyperplane can be expressed by a network in the way that I described. $\endgroup$ – zachdj Nov 7 at 20:39
  • $\begingroup$ To explain a bit more: The difference between hard-margin and soft-margin comes in at training time. But your question asks about translating an already-trained model to a neural network. $\endgroup$ – zachdj Nov 7 at 20:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.