0
$\begingroup$

Given that one wants to ascribe a class to groups of measurements using a classifier model, in what way can one include information about measurement accuracy?

More specifically, is there a feature engineering strategy for including information with the intent to allow the model to weight a subset of the sample more than other subsets?

As an example, say I have have coordinate and accuracy information (lat, lon, measurement_accuracy) in a time-series. I want to make a classifier which takes a group of 2 coordinate pairs, and output a class from this group. Each sample therefore consists of 2 coordinate pairs.

The measurement accuracy could be used to indicate to the model which of these measurements should be trusted most, but does not by itself give any hint towards the class the sample belongs to.

given the 3 samples

[ # learn to trust the first coordinate set more in making a prediction
    {'lat': 11, 'lon': 40, 'acc': 5},
    {'lat': 9, 'lon': 40, 'acc': 2}
],
[ # learn to trust the second coordinate set more in making a prediction
    {'lat': 74, 'lon': 131, 'acc': 1},
    {'lat': 78, 'lon': 140, 'acc': 4}
],
[ # learn to treat these coordinates approx. equally
    {'lat': 74, 'lon': 131, 'acc': 1},
    {'lat': 78, 'lon': 140, 'acc': 1}
]

and assuming acc is an indication of the coordinate pair measurement accuracy I would like the model to trust the first coordinate pair more in the first sample, trust the second one more in the second sample, and in the third sample trust both coordinate pairs equally.

What kind of a representation of the data could enforce this logic, given that the data will be fed into the model in a tabular way?

$\endgroup$
3
  • $\begingroup$ Usually one incorporates this sort of information in the loss function. You generally need to know a little more about what is meant by "accuracy". A good rule of thumb is that points with higher accuracy should be weighted more highly in your loss function and low accuracy points vice-versa. This interpretation is generally easier to justify if the accuracy relates to the target variable rather than the features $\endgroup$ – gazza89 Aug 26 '20 at 12:52
  • $\begingroup$ @gazza89 I absolutely agree with your point on if it were a target it makes sense to put it in your loss function. But this does not enable a model to learn that the accuracy of one measurement is better than another, only that it is less penalized for failing at low accuracy. If I understand you correctly, you are saying that for any given sample I should weight the loss according to something like the maximum or mean of all accuracies within that sample? If so, then I can't see how this would enable the model to prioritize a set of coordinates over another in making a prediction. $\endgroup$ – M.T Aug 26 '20 at 14:27
  • $\begingroup$ hmmm.. Maybe you could train a neural network (/two output model) which outputs both the mean and the prediction variance (if you were doing regression) and hope that the network learns that the "acc" parameter is predictive of high prediction variance. I think I agree with you that when the accuracy refers to the features, it's less obvious whether this should translate to downweighting in the loss function although is still feels intuitively right $\endgroup$ – gazza89 Aug 26 '20 at 15:50
0
$\begingroup$

You could use a multi label classifier so that your model could use the extra feature to decrease its confidence score. If there is indeed a correlation with inaccuracy of the measurement and "acc" it should be able to account for that in its prediction through its probability output

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.