I am using machine learning regression models to predict motor scores among a population with spinal cord injury using features derived from their actual movements. Although the clinical measure we are predicting is an ordinal integer from 0-5, we want to predict a continuous values between 0-5 to provide greater resolution of scores, but still produce an interpretable outcome for clinicians. Since the clinical score we are comparing to is not super reliable and is only measured in integers, it isn't a great "ground truth" to predict although it is used incredibly frequently in clinical settings. However, any other measures of strength are expensive to measure and may still not be that reliable, so there isn't a great other option for a ground truth for strength in this population.
The problem I am encountering is how to quantify the accuracy/error of our model? I want to relate our measure to the most commonly used clinical measure for interpretability, but I think if we report mean squared errors or anything similar we will inflate our error when the real issue is not that our measure is inaccurate but that the clinical measure we are comparing to is inaccurate or not specific enough. For example, if we predict a score of 4.4 when the clinical measure score was 4, how do we know if our prediction was off by .4 or whether the clinical measure just wasn't specific enough to know the difference between 4.4 and .4 and our measure is?
Should I round off our predictions to whole numbers for the sake of comparison to the clinical measure? Should I report the correlation between the predicted and actual instead of mean errors? Thank you for any help or suggestions!