I'm reading page 495 of Elements of Statistical Learning.

Here we discuss a technique for transforming the density estimation problem into one of supervised function approximation. This forms the basis for the generalized association rules described in the next section.

Let $g(x)$ be the unknown data probability density to be estimated, and $g_0(x)$ be a specified probability density function used for reference. For example, $g_0(x)$ might be the uniform density over the range of the variables. Other possibilities are discussed below. The data set $x_1, x_2, . . . , x_N$ is presumed to be an i.i.d. random sample drawn from $g(x)$. A sample of size $N_0$ can be drawn from $g_0(x)$ using Monte Carlo methods. Pooling these two data sets, and assigning mass $w = N_0/(N + N_0)$ to those drawn from $g(x)$, and $w_0 = N/(N + N_0)$ to those drawn from $g_0(x)$, results in a random sample drawn from the mixture density $(g(x) + g_0(x)) /2$. If one assigns the value $Y = 1$ to each sample point drawn from $g(x)$ and $Y = 0$ those drawn from $g_0(x)$, then $$μ(x) = E(Y | x) =\frac{g(x)} {g(x) + g_0(x)}$$

can be estimated by supervised learning using the combined sample $(y_1, x_1), (y_2, x_2), . . . , (y_{N+N_0}, x_{N+N_0})$ as training data.

I don't understand what are mass and mixture density. And I want to know why $μ(x) = E(Y | x) = \frac{g(x)}{g(x) + g_0(x)}$.


1 Answer 1


Fascinating problem, welcome to the site! In the second edition, this is covered in section 14.2.4 Unsupervised as Supervised Learning. The idea here is to design a classifier (that's supervised learning) between the desired but unknown distribution, and a known reference distribution (14.10), then to use this classifier along with the reference distribution to estimate the unknown distribution at any point (14.12). You can see that when the classifier deems a point to have come from $g$ ($\hat \mu \to 1$), the density $g_0 \hat \mu /(1-\hat \mu)$ increases. The converse is also true. This is what you would expect.

Mass refers to the probability measure of the sample. The idea is to redress by difference in sample sizes by weighting the probability of each sample inversely by its size (or equivalently, the size of the other sample). That's why the density of the mixture is the average of the components. A mixture density is simply a distribution formed by a mixture or sum (of two distributions in this case). Finally, $\mu$ is commonly used in statistics to denote the mean. The conditional mean of $Y$ is simply the probability that $x$ is drawn from $g$ (because the $Y=0$ term does not contribute to the expectation), thus

$$\mu(x) \equiv \mathbb E(Y|x) = \mathrm P(Y=1|x) = \frac{g}{g+g_0}(x)$$

This equation comes from Bayes' rule, after using $g = P(x|Y=1)$ and $g_0 = P(x|Y=0)$. This approach shows the power of using density ratios. In fact, there's a whole book devoted it.

  • $\begingroup$ But the mass are N/(N+N0) ( those points from g0(x) ), what does it mean? What have the probability N/(N+N0) ? $\endgroup$
    – Will Cai
    Aug 7, 2017 at 11:53
  • $\begingroup$ And i also want to ask what's a random sample of a density? $\endgroup$
    – Will Cai
    Aug 7, 2017 at 12:22
  • $\begingroup$ It means that the weight associated with each point should be thus. A sample of a density is simply a collection of draws (random numbers) from that density. In figure 14.3, the amber dots are sampled from $g_0$ while the blue dots are sampled from $g$. I think this is an interesting technique you should try for yourself; it's fairly straightforward. $\endgroup$
    – Emre
    Aug 7, 2017 at 15:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.