1
$\begingroup$

I have the following challenge and I very much hope that there is a solution to it. I also suspect that there is a simple approach to it. I just don't see it at the moment. Any help or advice is highly appreciated.

So, I have the following situation:

I asked persons to label about 1000 data points (each twice) on a 5-point scale, whose scores are not equi-distant. Texts were assessed with regard to several qualitative characteristics (such as comprehensibility). As was to be expected, the labelers did not always agree on the assessment. By analysing the inter-rater reliability, however, a "substantial" reliability (according to Landis and Koch) could be determined.

Now I want to use the labelled data as input for a machine learning algorithm (e.g. SVM and Random Forest). The challenge now is how to optimize the data in advance. Currently it is the case that for the same sample there are also different labels available.

The average value between different labels does not seem reasonable to me. So are there standard procedures how I can adjust the data set in advance?

You would help me a lot!

Thanks a lot in advance.

$\endgroup$
0
$\begingroup$

If you intend to use a summary statistics you would engineer it so it is well suited for your task, meaning captures most of the relevant information. For these things there is usually no best universal solution but it is problem specific. You did not specify what your problem is about so I can't help you there much, maybe use the median value.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you, @bonfab, for your answer. I have had texts assessed by two experts regarding e.g. comprehensibility on a scale of 1-5. These assessments are naturally different. I would like to use this labeled data to predetermine the comprehensibility of unlabeled texts by means of individual features (e.g. text length, #words) with a Random Forest algorithm. The problem is that I now have multiple labels for a multi-class problem. One expert assessed with 2, the other with 5. Please correct me if I am wrong. But I have always understood that the training set/ground truth should be unique. $\endgroup$ – requalys May 30 at 18:16
  • $\begingroup$ You can also train for multiclass, by example summing up several loss functions. However in your case you don't have a multiclass problem because for multiclass problems the classes usually don't exclude each other. In your case there seems to be ambiguity on how to interpret the texts. Maybe you should restructure the labelling process, such that there is as little ambiguity as possible, like asking your experts to reach a consens on their decisions. $\endgroup$ – bonfab May 30 at 20:42
  • $\begingroup$ Thanks, bonfab. Is there any approach to use the already labeled data for a classification task? Does some kind of standard practice exist in case of different labelings for a multi-class problem? As I mentioned, I dont think that it is helpful, just to delete the outlier or take the average of two differentiating labels. $\endgroup$ – requalys Jun 4 at 10:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.