Assume a continuous dependent variable $Y$. Then, $\hat{Y}$ is the predictions made by a machine learning-based model using a set of independent variables $X$.
In this case, however, it is known that $Y$ is not the best possible values for this variable. That is, $Y$ is actually the sum of $Y^{true}$ and $\epsilon$. $Y^{true}$ is the hidden true value of the variable that cannot be known to anyone. $\epsilon$ does not follow a known distribution or pattern.
In this case, any error measure $f(Y, \hat{Y}$) is not trusted. In principle, it is believed that human experts can judge the quality of predictions $\hat{Y}$ by examining the values of $X$. However, there are many ways where bias can sneak into this evaluation and it might be difficult to quantify the prediction quality.
Is there an established methodology (i.e., in terms of number of experts, the followed procedures, on which scale the prediction quality is judged, how conflicting expert opinions are accounted for, etc.) for evaluating the quality of such predictions?
If it makes it easy to understand or answer, the data and problem can be assumed to be in a business domain.
