1
$\begingroup$

Currently, I am working on a project. The dataset is balanced roughly in the ratio of 50:50. I created a decision tree classifier. I am achieving decent accuracy (~75%) on validation data but the precision for the target variable is biased. For class=0 it is approx. 98% while for the class = 1 it is just 17%.

I have tried scaling the data using MinMaxScaler still no luck.

model = tree.DecisionTreeClassifier(class_weight={1:30}, min_samples_leaf=160, max_depth=10)

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.3, random_state=10)

min_max_scaler = preprocessing.MinMaxScaler()
X_train_scaled = min_max_scaler.fit_transform(X_train)
X_test_scaled = min_max_scaler.fit_transform(X_test)

model = model.fit(X_train_scaled, y_train)

prediction = model.predict(X_test_scaled)

print metrics.accuracy_score(y_test, prediction)
print classification_report(y_test, prediction)
Size of x_train_scaled = 12600 and x_test_scaled = 5400
Accuracy: 75%
Precision: {0:100%, 1:17%}
Recall: {0:74%, 1:100%}
F1-Score: {0:85%, 1:29%}

How can I improve the precision of class=1 while still maintaining the overall precision and accuracy?

$\endgroup$
  • $\begingroup$ are your features numerical? $\endgroup$ – Media Feb 18 '18 at 9:24
  • $\begingroup$ There are a few numerical and a few categorical features $\endgroup$ – user3425778 Feb 18 '18 at 9:25
  • $\begingroup$ Actually I've never seen scaling categorical features. $\endgroup$ – Media Feb 18 '18 at 9:26
5
$\begingroup$

I guess differences in accuracies between class 0 and class 1 come from the class_weight parameter you have used. Class 1 will benefit from this overweighting towards class 0. You could try to play on this parameter to re-balance your results in class 0 and class 1.

An other idea could be to play on probabilities outputs and decision boundary threshold. Remember than when calling for method .predict(), sklearn decision tree will compare outputed probability with threshold 0.5. If it is greater than 0.5, then it assign class 1. On the contrary, if it is less than 0.5, it will assign class 0. You could try to play on this threshold by outputing probabilities first with .predict_proba()[:,1] and then test results for different thresholds decision boundaries. You can operate such as below :

model = clf.fit(df[features], df[label])
df["proba"] = model.predict_proba(df[features])[:,1]
threshold = 0.4 # You can play on this value (default is 0.5)
df["pred"] = df["proba"].apply(lambda el: 1.0 if el >= threshold else 0.0)
$\endgroup$
  • $\begingroup$ Thanks for the answer. I got a good amount of improvement by changing the class_weights. Now I also want to try to change the threshold but I am not finding a way to do that. Could you please tell me how to do that in sklearn? $\endgroup$ – user3425778 Feb 19 '18 at 19:33
  • $\begingroup$ I have edited answer with possible instructions. $\endgroup$ – Theudbald Feb 21 '18 at 10:19
0
$\begingroup$

While adjusting the Probability threshold, care must be taken that we use the predictions on the train data to do so. If we adjust the threshold based on the predictions made on the test data, we will be simply overfitting the model on the test data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.