1
$\begingroup$

I have a dataset belonging to three different classes: A, B and C. Among these three classes, the classification for label C is unreliable comparing to other two classes. In other words, some of the samples in class C is actually belong to class A and class B. For now, I need to run some supervised learning (logistic regression, decision tree and random forests) models. According to the confusion matrix, the classification between A and B is relatively accurate, but the classification between C and other two classes is not acceptable. I wondered whether is any way to deal with this issue?

For now, I'm considering to use a clustering algorithm for the samples in class C before runnning the model. After dividing the samples in class C into 3 groups, try to find a relatively better group as the datasets for class C.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

You have a common issue in which your learning data is noisy. Of course if you can do anything to make your data better/use more data this is the way to go. But I assume manual cleaning at large is not possible.

Reasons for the noise?

There are different approaches in machine learning to deal with noisy data. It depends a lot on where your noise is coming from. It helps to understand why your labels are incorrect. In your case it sounds like there is a structural mistake where you systematically misclassify under certain conditions.

ML methods to solve the noisy data issue

Besides simple rules (if applicable) there are more complex ways to approach the issue. There are many papers (e.g. this, this and this) that suggest different variants of the following:

  1. Manually create a small subset of cleaned labels
  2. Train the network on clean and noisy labels
  3. Use an additional network (or layer) to either create a mapping from noisy to clean labels or to learn about the noise distribution

This way you try to make the network recognizing its own mistake. Some of these models are rather complex, others are quite complex. There is even a paper that trains the network to also classify the noise and than make predictions based on features + noise classification.

I am not sure if there are any best practises yet which architecture works best in which case. You may need to try out some alternatives.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Thanks very much! I will check these papers. Besides that, as you said at the beginning, there did exists some systematical misclassfications for the datasets due to the technical issues in our database. I have talked to the people who are more familiar with this database, and he also provided me some other sources of data to improve the data quality in my dataset. Hope it works. $\endgroup$
    – JoshuaW1990
    Mar 6, 2018 at 16:52
  • $\begingroup$ Awesome! If you get more data and you have enough samples your network will probably pick up the pattern that the additional data is related to classify C. Let me know how things are going, especially if you decide to apply some noisiness classification. $\endgroup$
    – Gegenwind
    Mar 6, 2018 at 17:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.