Decision tree classifier: possible overfitting

Question

I have a dataset with following specifications:

• Training dataset with 52968 samples with 8562 positives
• Test Dataset with 13242 samples with 2135 positives
• There are 137 features

I want to perform a binary classification. I create DecisionTreeClassificator in pipeline:

imp = Imputer(strategy="most_frequent", axis=0)
var_thr = VarianceThreshold(threshold=1.7)
pca = RandomizedPCA(n_components=16)
clf = DecisionTreeClassifier(max_features=0.86, max_depth=42)

return Pipeline(steps=[('imp', imp),
                       ('var_thr', var_thr),
                       ('pca', pca),
                       ('clf', clf)
])

I also tried to increase training data with positive results:

series = y_train[y_train==1]
dupli = x_train.loc[series.index.tolist(), :]
for _ in range(5):
    x_train = x_train.append(dupli)
    y_train = y_train.append(series)

return x_train, y_train

After fitting my model, the score result for my test data is 0.9954, and the cross validation is:

cross_val_score(clf, x_train, y_train, cv=5)
[ 0.90225866  0.90638078  0.90592215  0.90007453  0.90632345]

Classification report for training data is perfect:

             precision    recall  f1-score   support

          0       1.00      1.00      1.00     44406
          1       1.00      1.00      1.00     42810

avg / total       1.00      1.00      1.00     87216

The confusion matrix is:

[[44203   203]
 [  190 42620]]

but the test data is much worse:

             precision    recall  f1-score   support

          0       0.85      0.85      0.85     11107
          1       0.21      0.21      0.21      2135

avg / total       0.75      0.75      0.75     13242

Confusion matrix is:

[[9428 1679]
 [1687  448]]

I used GridSearchCV for the threshold, n_components, max_features and max_depth. How can I improve my model and obtain better prediction?

EDIT -------> I changed clf in pipeline. I used RandomForestClassifier.

clf = RandomForestClassifier(
    n_estimators=500, n_jobs=-1, max_features=0.5, max_depth=15, 
    random_state=1
)

Now cross validation is

[ 0.81552396  0.81218827  0.82331021  0.81488276  0.81769191]

Classification report for training data with confusion matrix:

score train result: 0.8514148780040359
             precision    recall  f1-score   support

          0       0.86      0.85      0.85     44406
          1       0.85      0.85      0.85     42810

avg / total       0.85      0.85      0.85     87216

[[37757  6649]
 [ 6310 36500]]

Classification report for test data with confusion matrix:

score test result: 0.7341791270200876
             precision    recall  f1-score   support

          0       0.89      0.79      0.83     11107
          1       0.30      0.47      0.36      2135

avg / total       0.79      0.73      0.76     13242

[[8719 2388]
 [1132 1003]]

It looks better but I search model with + 0.90 recall for traning and test data sets.

Answer 1

Perfect training AUC is the hallmark of overfitting. When searching parameters, I often use a combination of test set AUC and the difference between training AUC and test AUC. When just using test set AUC in the search loss function, I found the model to still be pretty sensitive to which random set of test rows it sampled.

I also agree with the commenters that you should get better results with much smaller max_depth, probably on the order of 1-10, and ensemble models. Random forests are an excellent place to start.