I have a binary classification problem, which I am solving using Scikit's RandomForestClassifier. When I plotted the (by far) most important features, as boxplots, to see if I have outliers in them, I found many outliers. So I tried to delete them from the dataset.
The accuracy and Cross-Validation dropped by approximately 5%. I had 80% accuracy and an Cross-Val-Score of 0.8
After removing the outliers from the 3 most important_features (RF's feature_importance) the accuracy and Cross-Val-Score dropped to 76% and 77% respectively.
Here is a part of the description of my dataset:
Here is an overview of my data:
Here are the boxplots before removing the outliers:
Here are the feature importances before removing outliers:
Here is the accuracy and Cross-Val-Score:
Accuracy score: 0.808388941849 Average Cross-Val-Score: 0.80710845698
Here is how I removed the outliers:
clean_model = basic_df.copy() print('Clean model shape (before clearing out outliers): ', clean_model.shape) # Drop 'num_likes' outliers clean_model.drop(clean_model[clean_model.num_likes > (1938 + (1.5* (1938-125)))].index, inplace=True) print('Clean model shape (after clearing out "num_likes" outliers): ', clean_model.shape) # Drop 'num_shares' outliers clean_model.drop(clean_model[clean_model.num_shares > (102 + (1.5* (102-6)))].index, inplace=True) print('Clean model shape (after clearing out "num_shares" outliers): ', clean_model.shape) # Drop 'num_comments' outliers clean_model.drop(clean_model[clean_model.num_comments > (54 + (1.5* (54-6)))].index, inplace=True) print('Clean model shape (after clearing out "num_comments" outliers): ', clean_model.shape)
Here are the shapes after removing the outliers:
Clean model shape (before clearing out outliers): (6992, 20) Clean model shape (after clearing out "num_likes" outliers): (6282, 20) Clean model shape (after clearing out "num_shares" outliers): (6024, 20) Clean model shape (after clearing out "num_comments" outliers): (5744, 20)
Here are the boxplots after removing the outliers (still have outliers somehow.. If I delete these too, I will have really few datapoints):
Here is the accuracy and Cross-Val-Score after removing the outliers and using same model:
Accuracy score: 0.767981438515 Average Cross-Val-Score: 0.779092230906
How come is removing the outliers drops the accuracy and F1-score? Should I just leave them in the dataset? Or remove the outliers that are to see in the 2nd boxplot (after removing the 1st outliers as shown above)?
Here is my model:
model= RandomForestClassifier(n_estimators=120, criterion='entropy', max_depth=7, min_samples_split=2, #max_depth=None, min_samples_split=2, min_samples_leaf=1, min_weight_fraction_leaf=0.0, max_features=8, max_leaf_nodes=None, min_impurity_decrease=0.0, min_impurity_split=None, bootstrap=True, oob_score=False, n_jobs=1, verbose=0, warm_start=False, class_weight=None, random_state=23) model.fit(x_train, y_train) print('Accuracy score: ', model.score(x_test,y_test)) print('Average Cross-Validation-Score: ', np.mean(cross_val_score(model, x_train, y_train, cv=5))) # 5-Fold Cross validation
random_statewhich will give you the appearance that a result is stable, but by removing records, you essentially have a different random state and you should expect different performance measure. Cross validation will give you a better idea about the impact of removing the outliers. $\endgroup$