1
$\begingroup$

Assume a continuous dependent variable $Y$. Then, $\hat{Y}$ is the predictions made by a machine learning-based model using a set of independent variables $X$.

In this case, however, it is known that $Y$ is not the best possible values for this variable. That is, $Y$ is actually the sum of $Y^{true}$ and $\epsilon$. $Y^{true}$ is the hidden true value of the variable that cannot be known to anyone. $\epsilon$ does not follow a known distribution or pattern.

In this case, any error measure $f(Y, \hat{Y}$) is not trusted. In principle, it is believed that human experts can judge the quality of predictions $\hat{Y}$ by examining the values of $X$. However, there are many ways where bias can sneak into this evaluation and it might be difficult to quantify the prediction quality.

Is there an established methodology (i.e., in terms of number of experts, the followed procedures, on which scale the prediction quality is judged, how conflicting expert opinions are accounted for, etc.) for evaluating the quality of such predictions?

If it makes it easy to understand or answer, the data and problem can be assumed to be in a business domain.

$\endgroup$
1
$\begingroup$

Measuring the inter-annotator agreement is one of the ways to measure the reliability of the annotations.

(I assume that there are also other methods but I'm not that knowledgeable about the topic)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.