4
$\begingroup$

I have a numeric variable (price) and it has a long tail in both training and test data sets. I found that if you remove the highest 1% of the value in both train and test data set for this variable, then the histogram of this variable in train and test data set looks pretty much the same. See the following figure. enter image description here

My question is: I still need to use the training data (with both features and labels) to make predictions on the test data (with features only). In this case, how should I deal with this feature variable? I was thinking about removing the top 1% data in both training and test data set, but as I still need to make predictions on that 1% test data, so this is not a good idea I guess. In this example, as the empirical distribution of this variable in both training and test data sets look the same before and after removing the "outlier", should we just leave this variable unchanged? Also, in general, how should we handle the outlier before we put the feature into the machine learning algorithm?

$\endgroup$
2
  • $\begingroup$ Do plot a scatter plot among your values and lets say a range of indices(your df's length) This will make it clearer.. $\endgroup$
    – Aditya
    Mar 7 '18 at 0:37
  • $\begingroup$ @Aditya Could you please elaborate a little bit.....should I plot the price variable against the indices of my df? $\endgroup$
    – KevinKim
    Mar 8 '18 at 2:35
3
$\begingroup$

Dealing with outliers requires knowledge about the outlier, the dataset and possibly domain knowledge. Given this, there are many options to handle outliers.

Without taking a look at your specific data, it could be that this outlier represents a total? Perhaps the data source you have included totals, which should be removed.

Generally, figuring out what to do with outliers requires investigating the outlier.

If the outlier is a data processing or entry error, it can generally be removed, or replaced with say, the mean (without the outlier).

If errors have been ruled out think about whether the outlier could be a legitimate value? There are no Right or Wrong answer.

Instead, documentation of your decisions is really important. I'd suggest running your models with the outlier, and without, comparing the results and document any decision to remove or transform the outlier.

Depending on your purpose, it should become relatively clear as to whether the outlier should be removed, eg if you're visualising the data, removing the outlier will likely have more a meaningful impact than retaining it.

$\endgroup$
0
$\begingroup$

Will remove the answer once the op sees it(please comment when you have done so)

Scatter Plot

plt.figure(figsize=(8,6))
plt.scatter(range(df.shape[0]), np.sort(df.loan_amount.values))
plt.xlabel('index', fontsize=12)
plt.ylabel('loan_amount', fontsize=12)
plt.title("Loan Amount Distribution")
plt.show()

Try Box plot Also..

It does really help to correctly predict the outlier ones. We remove them for aesthetic visualisation of the data. However We can choose to (not) include them in our modeling activity because this is what we want to correct or improve on...

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.