0
$\begingroup$

The XGBoost Classifier I built is consistently returning a f1 score of 0 and I am unable to fix this despite experimenting with various hyperparameters. The data is heavily imbalanced and hence I feel the model in trying to maximize accuracy is behaving like this . Even changing the eval_metrics to use "aucpr" had no effect. I have searched on Google quite a bit but am unable to find out how to increase the f1 score as either I get high precision or high recall but not both . Please let me know if I am missing something

This is the code I am using. The csv file has 100,000 rows and 2 numeric columns as the feature set and 1 response variable (0 or 1) hence no transformation was carried out .

df = pd.read_csv("Output\\SimulatedData.csv")
X = df.drop('Response', axis=1)
y = df["Response"]
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=25)

estimator_ = XGBClassifier(objective='binary:logistic',  eval_metric='aucpr')
estimator_.fit(X_train, y_train)
y_pred = estimator_.predict(X_test)
Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ precision, recall, f1 score depend on the cut-off value. Change your cut-off value you will get a different answer. I suggest tuning the cut-off value for your business problem if still want to use f1 score. Better is to use a proper scoring rule - fharrell.com/post/class-damage. Like Darkness says below, get more data. Does your business problem really only have 2 features? The file says simulateddata. Simulate more data. Personally I do not like smote and over/under sampling may not be needed. You did not mention the imbalance ratio. Every model does not need 50-50 split. $\endgroup$
    – Craig
    Nov 17, 2022 at 10:08
  • $\begingroup$ Please let me know how I can change the cut off value .The actual business problem has 100,000 rows and 40 columns. This data has been simulated with 2 columns to better understand why the model does not work. The imbalance ratio is 93:7 .Thanks in Advance $\endgroup$
    – J.Sriram
    Nov 18, 2022 at 3:01
  • $\begingroup$ predict_proba() gets the probability. Then the cutoff value is optimized for the problem. Usually take in benefits/revenue of TP & TN vs costs of FP & FN. Or some other optimization or business problem metric. 93-7 is not heavily imbalanced. I typically work with models in the 99.99 to 0.01 and less. If I rebalance, I do not go above 90-10. But when there is a 99.99 to 0.01 I might go to 99-1. Instead of focusing on imbalance and hyperparameters, I recommend to focus on the data and cutoff with data priority. Maybe the data is not separable or you need more. $\endgroup$
    – Craig
    Nov 18, 2022 at 11:39
  • $\begingroup$ Using predict_proba() and a proper scoring rule as described by the link you shared has solved my issue . Thanks for your help. Please add it as an answer instead of a comment so that I can close this question $\endgroup$
    – J.Sriram
    Nov 19, 2022 at 5:18

3 Answers 3

Reset to default
1
$\begingroup$

I see you are more concentrated on trying to hyper optimize the model than actually looking into and solving the real problem here, which in my opinion is the data. It's almost always the data.

My solution would be to try and get more samples in a synthetic or non-synthetic way. Have you tried implementing any over-sampling techniques such as SMOTE or ADASYN and then running XGBoost?

Improve this answer
$\endgroup$
1
  • $\begingroup$ the problem I feel is the imbalance in the data but since it can't be changed I was trying to change the model . Yes , I did try smote and adasyn but it did not result in any significant improvement . Nor did specifying scale_pos_weight . $\endgroup$
    – J.Sriram
    Nov 18, 2022 at 3:08
1
$\begingroup$

Setting eval_metric doesn't change the learning behavior except for early stopping.

[...] the model in trying to maximize accuracy [...]

This is not what the model tries to do: it is optimizing the log-loss.

Tuning the decision threshold is probably the first thing to try; there is a (not exact) tradeoff between precision and recall as you move that threshold, so your experience in seeing a good score for one or the other means somewhere in between should be something more suitable.

Setting class weights will produce roughly the same results as changing the decision threshold, and might be easier especially in the context of a hyperparameter search.

Improve this answer
$\endgroup$
0
$\begingroup$

The answer which has been provided by Craig as a comment above is to use predict_proba() to get the probability and then to use a proper scoring rule.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.