2
$\begingroup$

I have a dataset with following specifications:

  • Training dataset with 52968 samples with 8562 positives
  • Test Dataset with 13242 samples with 2135 positives
  • There are 137 features

I want to perform a binary classification. I create DecisionTreeClassificator in pipeline:

imp = Imputer(strategy="most_frequent", axis=0)
var_thr = VarianceThreshold(threshold=1.7)
pca = RandomizedPCA(n_components=16)
clf = DecisionTreeClassifier(max_features=0.86, max_depth=42)

return Pipeline(steps=[('imp', imp),
                       ('var_thr', var_thr),
                       ('pca', pca),
                       ('clf', clf)
])

I also tried to increase training data with positive results:

series = y_train[y_train==1]
dupli = x_train.loc[series.index.tolist(), :]
for _ in range(5):
    x_train = x_train.append(dupli)
    y_train = y_train.append(series)

return x_train, y_train

After fitting my model, the score result for my test data is 0.9954, and the cross validation is:

cross_val_score(clf, x_train, y_train, cv=5)
[ 0.90225866  0.90638078  0.90592215  0.90007453  0.90632345]

Classification report for training data is perfect:

             precision    recall  f1-score   support

          0       1.00      1.00      1.00     44406
          1       1.00      1.00      1.00     42810

avg / total       1.00      1.00      1.00     87216

The confusion matrix is:

[[44203   203]
 [  190 42620]]

but the test data is much worse:

             precision    recall  f1-score   support

          0       0.85      0.85      0.85     11107
          1       0.21      0.21      0.21      2135

avg / total       0.75      0.75      0.75     13242

Confusion matrix is:

[[9428 1679]
 [1687  448]]

I used GridSearchCV for the threshold, n_components, max_features and max_depth. How can I improve my model and obtain better prediction?

EDIT -------> I changed clf in pipeline. I used RandomForestClassifier.

clf = RandomForestClassifier(
    n_estimators=500, n_jobs=-1, max_features=0.5, max_depth=15, 
    random_state=1
)

Now cross validation is

[ 0.81552396  0.81218827  0.82331021  0.81488276  0.81769191]

Classification report for training data with confusion matrix:

score train result: 0.8514148780040359
             precision    recall  f1-score   support

          0       0.86      0.85      0.85     44406
          1       0.85      0.85      0.85     42810

avg / total       0.85      0.85      0.85     87216

[[37757  6649]
 [ 6310 36500]]

Classification report for test data with confusion matrix:

score test result: 0.7341791270200876
             precision    recall  f1-score   support

          0       0.89      0.79      0.83     11107
          1       0.30      0.47      0.36      2135

avg / total       0.79      0.73      0.76     13242

[[8719 2388]
 [1132 1003]]

It looks better but I search model with + 0.90 recall for traning and test data sets.

$\endgroup$
  • 2
    $\begingroup$ It's possible that your test data and train data are presenting a different 'story'. What if your try shuffling all your data and then cross-validating. StratifiedShuffleSplit will preserve the ration of positives. You can run cross_val_score on all your data. $\endgroup$ – Hobbes Nov 2 '17 at 18:35
  • $\begingroup$ I tried StratifiedShuffleSplit, but result is the same. Recall for positive in test set still is ~0.24 $\endgroup$ – CezarySzulc Nov 2 '17 at 20:00
  • 2
    $\begingroup$ Decision trees are known for overfitting data. They grow until they explain all data. I noticed you have used max_depth=42 to pre-prune your tree and overcome that. But that value is sill too high. Try smaller values. Alternatively, use random forests with 100 or more trees. $\endgroup$ – Ricardo Cruz Nov 2 '17 at 21:21
  • 1
    $\begingroup$ There are a lot of things going on here. There are too many negatives. Try oversampling. Also, Decision Trees are prone to overfiitng. Try to use ensembles. And as @RicardoCruz said, max_depth is way too much. Typical values of Max_depth should be between 6-14 $\endgroup$ – enterML Nov 3 '17 at 10:35
1
$\begingroup$

Perfect training AUC is the hallmark of overfitting. When searching parameters, I often use a combination of test set AUC and the difference between training AUC and test AUC. When just using test set AUC in the search loss function, I found the model to still be pretty sensitive to which random set of test rows it sampled.

I also agree with the commenters that you should get better results with much smaller max_depth, probably on the order of 1-10, and ensemble models. Random forests are an excellent place to start.

| improve this answer | |
$\endgroup$
  • $\begingroup$ I try RandomForests and it's look nice but it have too low recall, for traning set it's 0.85 for positive and negative (it's ok) but for test set it's 0.79 for negative and 0.47 for positive. I used RandomForests with params: n_estimators=500, max_features=0.5, max_depth=15. $\endgroup$ – CezarySzulc Nov 5 '17 at 13:14
  • $\begingroup$ Your model will prone to over fit if you don't give learning rate/shrinkage parameter while using random forest. Use of CV and other techniques like grid search you can get optimum value of learning rate and should give the same while training the model $\endgroup$ – CodeMaster GoGo Oct 8 '18 at 10:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.