Does the Bayesian MAP give a probability distribution over unseen data?

Question

I'm working my way through the Bayesian world. So far I've understood that the MLE or the MPA are point estimates, therefore using such models just outputs one specific value and not a distribution.

Moreover, vanilla neural networks do in fact something like MLE, because minimizing the squared-loss or the cross-entropy is similar to finding parameters that maximize the likelihood. Furthermore, using neural networks with regularization is comparable to the MAP estimates, as the prior works like the penalty term in error functions.

However, I've found this work. It shows that the weights $W_{PLS}$ gained from a penalized least-squared are the same as the weights $W_{MAP}$ gained through Maximum a posterior:

However the paper says:

"The first two approaches result in similar predictions, although the MAP Bayesian model does give a probability distribution for $t_*$ The mean of this distribution is the same as that of the classical predictor $y(x_*; W_{PLS})$, since $W_{PLS} = W_{MAP}$

What I don't get here is how can the MAP Bayesian give a probability distribution over $t_*$, when it is only a point estimate? Consider a neural network - a point estimate would mean some fixed weights so how can there be a output probability distribution? I thought that this is only achieved in the True Bayesian where we integrate out the unknown weights, therefore building something like the weight averaged of all outcomes, using all possible weights.

Answer 1

Bayesian MAP gives a point estimate $\mathbf{w}_{MAP}$ for model parameter $\mathbf{w}$, which implies

$p(\mathbf{w}|\mathbf{t},\cdots) = \left\{\begin{matrix} 1 & \mathbf{w}=\mathbf{w}_{MAP}\\ 0 &\text{o.w.} \end{matrix}\right.$

However, distribution $p(t_*|x_*, \mathbf{w}_{MAP}, \sigma^2)$ is the probabilistic model of target $t_*$ given input $x_*$ and parameters; after eq. (8), paper explicitly says $x$'s are omitted from the givens of target likelihood $p(t|.)$, but I think they should have been kept there to avoid confusion. It might also be helpful to remind that $(\mathbf{x}, \mathbf{t}) = \{(x_1, t_1),...,(x_n, t_n)\}$ are the observed (input, target) pairs.

Paper says when $\mathbf{w}_{MAP}$ is calculated, which would be the same as $\mathbf{w}_{PLS}$, mean of target distribution will be the same as the deterministic target $t_* = y(x_*; \mathbf{w}_{PLS})$ from Penalized Least Squares. That is,

$\mathbb{E}[t|x_*,\mathbf{w}_{MAP}, \sigma^2] = \int tp(t|x_*,\mathbf{w}_{MAP}, \sigma^2)dt = y(x_*; \mathbf{w}_{PLS})$

In summary, in (Penalized) Least Squares, both target and parameters are deterministic. In non-Bayesian and Bayesian MAP, parameter is deterministic but target is probabilistic. In (true) Bayesian, both target and parameters are probabilistic. In full Bayesian, hyper-parameters (parameters of distribution of parameters, e.g. $\alpha$) are also probabilistic.