Replacing missing value by class conditional mean

Question

I have two classes, $p(x|y=0)$ and $p(x|y=1)$ with ${{\mu }_{0}}$ and ${{\mu }_{1}}$ as mean and shared covariance matrix $\Sigma $. Now, I have a missing feature ${{x}_{n}}$ for a particular observation. We replace this missing value by the class conditional mean $E({{x}_{n}}|y=0)$ and $E({{x}_{n}}|y=1)$. How do I justify this missing value treatment.

This is actually an exercise. But, something that I noticed is that if $p(x|y=0)$ and $p(x|y=1)$ have means ${{\mu }_{0}}$ and ${{\mu }_{1}}$, then how is their covariance matrix the same?

Assuming their covariance matrix is same, the only advantage I see is that if I take class conditional mean as missing value then in multivariate Gaussian, the effect of this is that the $n^{th}$ row of $\left( x-\mu \right)$ vector will be $0$ since $\left( {{x}_{n}}-{{\mu }_{n}} \right)$ will be $0$, thus making no impact in calculating $p(y=0|x)$ and $p(y=1|x)$.

Is this explanation correct?

Answer 1

Your intuition about 'no effect' is true in some sense. But this replacement may be not the best use of the information you have.

The choice of missing value treatment depends on your initial problem statement. In all the cases I assume that you have already somehow estimated conditional means $\mu_0$ and $\mu_1$ and the common variance matrix $S$.

Problem 1. For your test observation, you know $y$, but know nothing about $x$, and want to estimate $x_n$. If this is the case, $\mu_{y, n}$ is indeed your best guess of $x_n$.

Problem 2. For your test observation, you know $x_{-n}$ (that is, $x$ except $x_n$) and $y$, and want to predict $x_n$ as best as possible. If it is the case, you should use expectation conditional not only on $y$, but also on $x_{-n}$: $$ \hat{x_n} = \mathbb{E}(x_n|y, x_{-n}) = \mu_{(y,n)} + S_{(n,-n)}S_{(-n)}^{-1}(x_{-n}-\mu_{(y,-n)}) $$ where $S_{*}$, $\mu_{y,*}$ are corresponding parts of covariance matrix and conditional expectation vector.

Problem 3. You know $x_{-n}$ but don't know $y$, and you want to estimate $x_n$. In this case, conditional distribution of $x_n$ is a mixture of two normal distributions, $p(x_n|y=0, x_{-n})$ and $p(x_n|y=1, x_{-n})$, with weights, proportional to $p(y=0|x_{-n})$ and $p(y=1|x_{-n})$ respectively. The mean of this distribution is just the weighted average of two means conditional on $y$. But this distribution may be bimodal, and its mean may be not a very useful statistic.

Problem 4. You know only $x_{-n}$, and your final goal is to estimate probability of $y=1$. In this case, you should just forget about $x_n$ (you don't know anything of it anyway), and use the distribution of $x_{-n}$ conditional on $y$ for whatever algorithm you apply.