1
$\begingroup$

Gaussian kernel

$k(x,y) = \exp(-\lVert x-y \rVert^2/\sigma^2)$

has a hyperparameter $\sigma$.

I know grid search cross validation, but this would require a lot of computation since computational cost of kernel method scales with the number of samples to the power of 2.

$\endgroup$
1
  • $\begingroup$ Perhaps sisualize the kernel values for different bandwidth values. Compare to your dataset $\endgroup$ May 17, 2018 at 1:45

2 Answers 2

1
$\begingroup$

As shown in wikipedia for KDE, a rule-of-thumb bandwidth estimator can be given if the underlying density for your data is Gaussian. This estimator is given by: $h = (\frac{4\hat{\sigma}}{3n})^{1/5}$, where $h$ is the bandwidth of your KDE estimation, $n$ the number of data and $\hat{\sigma}$ the estimation of the standard deviation of your sample.

If the underlying distribution for your data is not gaussian, you can still try with this bandwidth, but it might smooth everything too much. In that case, you should go for cross-validation with smaller bandwidths.

$\endgroup$
1
  • $\begingroup$ I think your answer is not applicable here. The Gaussian kernel in KDE is equivalent to the standard normal distribution. The stated kernel function in the question however is not even a density. In contrast, it is the radial basis function which is also sometimes called Gaussian kernel in the context of e.g. SVMs. Then, you would rather use grid search, random search or other more advance hyperparameter optimization procedures. $\endgroup$
    – stats_guy
    Feb 9, 2019 at 14:05
1
$\begingroup$

After searching a few, I find this problem is really simple actually in some circumestances.

In Gausiann process regression, one can simply refer to Automatic relevance determination (ARD), which is to optimize the hyperparameters in kernels and also the observation noise variance, by maximize the marginal likelihood, using gradient descent!

This has been pretty standard and mature.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.