I have the following challenge and I very much hope that there is a solution to it. I also suspect that there is a simple approach to it. I just don't see it at the moment. Any help or advice is highly appreciated.

So, I have the following situation:

I asked persons to label about 1000 data points (each twice) on a 5-point scale, whose scores are not equi-distant. Texts were assessed with regard to several qualitative characteristics (such as comprehensibility). As was to be expected, the labelers did not always agree on the assessment. By analysing the inter-rater reliability, however, a "substantial" reliability (according to Landis and Koch) could be determined.

Now I want to use the labelled data as input for a machine learning algorithm (e.g. SVM and Random Forest). The challenge now is how to optimize the data in advance. Currently it is the case that for the same sample there are also different labels available.

The average value between different labels does not seem reasonable to me. So are there standard procedures how I can adjust the data set in advance?

You would help me a lot!

Thanks a lot in advance.


2 Answers 2


If you intend to use a summary statistics you would engineer it so it is well suited for your task, meaning captures most of the relevant information. For these things there is usually no best universal solution but it is problem specific. You did not specify what your problem is about so I can't help you there much, maybe use the median value.

  • $\begingroup$ Thank you, @bonfab, for your answer. I have had texts assessed by two experts regarding e.g. comprehensibility on a scale of 1-5. These assessments are naturally different. I would like to use this labeled data to predetermine the comprehensibility of unlabeled texts by means of individual features (e.g. text length, #words) with a Random Forest algorithm. The problem is that I now have multiple labels for a multi-class problem. One expert assessed with 2, the other with 5. Please correct me if I am wrong. But I have always understood that the training set/ground truth should be unique. $\endgroup$
    – requalys
    May 30, 2020 at 18:16
  • $\begingroup$ You can also train for multiclass, by example summing up several loss functions. However in your case you don't have a multiclass problem because for multiclass problems the classes usually don't exclude each other. In your case there seems to be ambiguity on how to interpret the texts. Maybe you should restructure the labelling process, such that there is as little ambiguity as possible, like asking your experts to reach a consens on their decisions. $\endgroup$
    – bonfab
    May 30, 2020 at 20:42
  • $\begingroup$ Thanks, bonfab. Is there any approach to use the already labeled data for a classification task? Does some kind of standard practice exist in case of different labelings for a multi-class problem? As I mentioned, I dont think that it is helpful, just to delete the outlier or take the average of two differentiating labels. $\endgroup$
    – requalys
    Jun 4, 2020 at 10:49

I would try to avoid any summary statistics. When I've dealt with similar problems the best solution has been to keep each labeled text as an independent entry. So if the original dataset looks like

| text  | label |
| "foo" | 0     |
| "foo" | 5     |
| "foo" | 1     |
| "bar" | 3     |
| "bar" | 3     |
| "bar" | 3     |

instead of doing something like

| text  | label   |
| "foo" | 3       |
| "bar" | 3       |

I would use the original dataset. With this approach, you're leaving to the model the possibility to learn that some texts are more difficult to label than other, for instance, "foo" has 3 different labels, while "bar" was labeled the same by all the labelers.

Also, having different labels for the same input is not a problem, it only means that given your input it's impossible to have perfect predictions, and since your problem deals with human subjectivity it makes sense that your model can't get perfect accuracy.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.