Pattern Recognition - Kernel Density estimators 2.5.1

Question

Please refer page 122-123 of Pattern recognition and Machine Learning - Bishop. A few equations:

Density estimate: $$ p(\mathbf{x}) = \frac{K}{NV} \tag{2.246}$$ where $K$ = #points in: $N$ regions of volume $V$ each.

Kernel function: Number $K$ of points falling within a hypercube centered at origin: $$\begin{equation}k(\mathbf{u}) =\begin{cases} 1, \quad |u_i| \leq \frac{1}{2} \qquad i = 1,\dots,D,\\ 0, \quad otherwise \end{cases} \tag {2.247} \end{equation}$$

From 2.247, the quantity $k((\mathbf{x} - \mathbf{x_n})/h)$ will be one if the data point $\mathbf{x_n}$ lies inside a cube of side $h$ centered on $\mathbf{x}$, and zero otherwise. The total number of data points lying inside this cube will therefore be: $$K = \sum_{n=1}^{N} k \Big(\frac{\mathbf{x} - \mathbf{x_n}}{h}\Big) \tag{2.248}$$ Substituting this expression into (2.246) then gives the following result for the estimated density at $\mathbf{x}$ $$p(\mathbf{x}) = \frac{1}{N}\sum_{n=1}^{N} \frac{1}{h{^D}}k\Big( \frac{\mathbf{x} - \mathbf{x_n}}{h}\Big) \tag{2.249}$$

where we have used $V$ = $h^D$ for the volume of hypercube of side $h$ in $D$ dimensions. Using the symmetry function $k(\mathbf{u})$, we can now re-interpret this equation, not as a single cube centered on $\mathbf{x}$ but as the sum over $N$ cubes centered on $N$ data points $\mathbf{x_n}$

I am struggling to follow explanation of last paragraph. Each point $\mathbf{x_n}$ is at the center of (respective) cube - total $N$ cubes. If so, then concept of boundary/ distance from a fixed point seems to be missing - ref. 2.247, 2.248 resulting all 1's. Which points will be zero, if any.

Just a rough sketch:

Left sketch represents 2.248 - imagine few points outside the cube (=0); points inside cube=1. Right sketch is my understanding of 2.249's explanation: 3 points, all at center of respective cube. On what basis will the data points be classified as 0 or 1?

Answer 1

Pls. refer the equations above. In case of eq 2.248 (left box image) we're calculating density at point $x$, only. There are total of 3 points, only 2 qualify as per kernel function and hence probability at $x$, $p(x) = \frac{2}{3}$.

The right box diagram shows that we take the points, one by one, and calculate the probability at each point using kernel function. Thus we get probability at each of point of data set. And effectively, we get a probability distribution for the entire data set.

The bar chart shows the PMF for 2 cases - left: probability based on (eq 2.248) and right: for (eq. 2.249). I use PMF to easily communicate the understanding.

*Note:

1- With x_1 centered cube 2 points: x_1 & x_2 lie inside the cube; same for x_2 centered cube. However for x_1 only one point x_1 lies in the cube - out of total of 3 points.

2- d|a, b| gives distance between points a, b.

3- for a large collection of data points and cubes of small sides (h), we get a PDF distribution curve.