0
$\begingroup$

So I have data that has some illogical entries, in the picture below you can see a person with 11 years in the company and he takes 100 dollars, this is for sure wrong and should be removed right?

The person under them has almost the same attributes but yet he gets 1800 dollars. The data that I'm showing in the picture is looking really bad right? At index 1479 the person has 16 years in the company yet he only gets 191 dollars.

Should I make some equation to remove those bad observations depending on Service Period and salary for example? Or should I keep them and they are normal? Will removing illogical observations like these improve my model?

enter image description here

Improve this question
$\endgroup$
4
  • $\begingroup$ What are you doing with the data? - Building a report for data quality, building a model that predicts salary, ...? Is the data wrong or rare? $\endgroup$
    – Craig
    Oct 6, 2021 at 9:29
  • $\begingroup$ @Craig building a model to predict the salary. $\endgroup$
    – Maxi
    Oct 6, 2021 at 9:34
  • 2
    $\begingroup$ This really requires domain knowledge, we cannot tell you what is wrong or right. You should communicate with whoever gave you this data and try to fix it manually. I would avoid automatic ways, or at least use them only as a screening tool, but each such "illogical" case should be treated separately, to make sure you don't bias your data. If you think they are wrong then I wouldn't include them in the model. $\endgroup$
    – user2974951
    Oct 6, 2021 at 11:05
  • $\begingroup$ @user2974951 Thank you for your answer $\endgroup$
    – Maxi
    Oct 6, 2021 at 11:09

2 Answers 2

Reset to default
2
$\begingroup$

Before removing anything from your dataset (nonsensical values or outliers), it is wise to have an opinion of a subject matter expert. It might be that the person with 11 years and earning 100 dollars might not be contributing to the company much and hence is not receiving a raise (hence a valid outlier!). Or it might be an error of the person who created the dataset (and hence a nonsensical value!).

Consult the person to see which of the above two cases does the entry fall in and then decide for yourself!

Improve this answer
$\endgroup$
1
  • $\begingroup$ thank you for your answer, I've already sent an email asking for an explanation about those values. $\endgroup$
    – Maxi
    Oct 6, 2021 at 12:04
1
$\begingroup$

From a ML perspective I see two possible approaches:

outlier detection

You can use a clustering algorithm like t-sne to visualize your data, and identify possible outliers. You can use this to understand the data, what outliers exist, and why. As @spectre suggested, you may want to bring this information to a subject matter expert.

this can also be where things end. You can simply say "throw out data if it's too much of an outlier" based on k-means output, for instance.

supervised classification

Once you have a solid profile of what is an issue and why, you can manually flag data that is problematic. You can then train a classification model like xgboost, for instance, to learn from what you classified as problematic, and predict if new entries are problematic.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for you answer! $\endgroup$
    – Maxi
    Oct 7, 2021 at 6:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.