Decision tree classifier: possible overfitting

I have a dataset with following specifications:

• Training dataset with 52968 samples with 8562 positives
• Test Dataset with 13242 samples with 2135 positives
• There are 137 features

I want to perform a binary classification. I create DecisionTreeClassificator in pipeline:

imp = Imputer(strategy="most_frequent", axis=0)
var_thr = VarianceThreshold(threshold=1.7)
pca = RandomizedPCA(n_components=16)
clf = DecisionTreeClassifier(max_features=0.86, max_depth=42)

return Pipeline(steps=[('imp', imp),
('var_thr', var_thr),
('pca', pca),
('clf', clf)
])


I also tried to increase training data with positive results:

series = y_train[y_train==1]
dupli = x_train.loc[series.index.tolist(), :]
for _ in range(5):
x_train = x_train.append(dupli)
y_train = y_train.append(series)

return x_train, y_train


After fitting my model, the score result for my test data is 0.9954, and the cross validation is:

cross_val_score(clf, x_train, y_train, cv=5)
[ 0.90225866  0.90638078  0.90592215  0.90007453  0.90632345]


Classification report for training data is perfect:

             precision    recall  f1-score   support

0       1.00      1.00      1.00     44406
1       1.00      1.00      1.00     42810

avg / total       1.00      1.00      1.00     87216


The confusion matrix is:

[[44203   203]
[  190 42620]]


but the test data is much worse:

             precision    recall  f1-score   support

0       0.85      0.85      0.85     11107
1       0.21      0.21      0.21      2135

avg / total       0.75      0.75      0.75     13242


Confusion matrix is:

[[9428 1679]
[1687  448]]


I used GridSearchCV for the threshold, n_components, max_features and max_depth. How can I improve my model and obtain better prediction?

EDIT -------> I changed clf in pipeline. I used RandomForestClassifier.

clf = RandomForestClassifier(
n_estimators=500, n_jobs=-1, max_features=0.5, max_depth=15,
random_state=1
)


Now cross validation is

[ 0.81552396  0.81218827  0.82331021  0.81488276  0.81769191]


Classification report for training data with confusion matrix:

score train result: 0.8514148780040359
precision    recall  f1-score   support

0       0.86      0.85      0.85     44406
1       0.85      0.85      0.85     42810

avg / total       0.85      0.85      0.85     87216

[[37757  6649]
[ 6310 36500]]


Classification report for test data with confusion matrix:

score test result: 0.7341791270200876
precision    recall  f1-score   support

0       0.89      0.79      0.83     11107
1       0.30      0.47      0.36      2135

avg / total       0.79      0.73      0.76     13242

[[8719 2388]
[1132 1003]]


It looks better but I search model with + 0.90 recall for traning and test data sets.

• It's possible that your test data and train data are presenting a different 'story'. What if your try shuffling all your data and then cross-validating. StratifiedShuffleSplit will preserve the ration of positives. You can run cross_val_score on all your data. Nov 2 '17 at 18:35
• I tried StratifiedShuffleSplit, but result is the same. Recall for positive in test set still is ~0.24 Nov 2 '17 at 20:00
• Decision trees are known for overfitting data. They grow until they explain all data. I noticed you have used max_depth=42 to pre-prune your tree and overcome that. But that value is sill too high. Try smaller values. Alternatively, use random forests with 100 or more trees. Nov 2 '17 at 21:21
• There are a lot of things going on here. There are too many negatives. Try oversampling. Also, Decision Trees are prone to overfiitng. Try to use ensembles. And as @RicardoCruz said, max_depth is way too much. Typical values of Max_depth should be between 6-14 Nov 3 '17 at 10:35