3
$\begingroup$

I read a recent paper by Pedro Domingos, claiming that Every Model Learned by Gradient Descent Is Approximately a Kernel Machine.

I wanted to understand the key idea a little better. Why are neural networks equivalent to kernel methods? And in particular, is there any method to actually derive the equivalent kernel from an existing neural network?

$\endgroup$
1

1 Answer 1

1
$\begingroup$

First of all, the paper only implies that under certain conditions, neural networks and kernel functions exhibit an equivalence.

This is because the neural networks with their hidden layers converge during the training process to some optimal weight values for each neuron. These values align with a kernel matrix which means that their representations can be approximated by a kernel function.

Take for example support vector machines (SVMs) where the boundary between classes is determined by the similarity/kernel function between the inputs. A neural network would learn this through complex non-linear transformations using their activation function.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.