1
$\begingroup$

I need to improve the prediction result of an algorithm that is already programmed based on logistic regression ( for binary classification).

I tried to use XGBoost and CatBoost (with default parameters). but it takes a long time to train the model (LR takes about 1min and boost takes about 20 min). and if I want to apply tuning parameters it could take more time for fitting parameters.

I want to ask if there are any suggestions to apply fastly boosting methods. And if there are other ways to get better performance I hope to mention them, please.

Ps: My data is about 280 000 simples and 247 (numerical) features;

Improve this question
$\endgroup$
2
$\begingroup$

XgBoost often does better than Logistic Regression. I would use CatBoost when I have a lot of categorical features or if I do not have the time for tuning hyperparameters.

You should invest time in a boosting model for sure (they will always take more time than Logistic Regression) because it is worth it. If you are impatient, try CatBoost instead of XgBoost to see the improvement in accuracy. That will definitely give you the motivation to spend more time tuning your boosted trees.

Improve this answer
$\endgroup$
  • $\begingroup$ Thanks Garima for your help, what about Stochastic gradient descent vs xgboost or catboost $\endgroup$ – Nirmine Feb 16 '19 at 16:30
  • $\begingroup$ Stochastic gradient descent is just an algorithm to update parameters in an optimum way. It is used with other algorithms like linear regression, neural networks, etc. (not limited to these though). However, it is not used in gradient boosted trees. So you cannot compare it with XgBoost and Catboost. $\endgroup$ – Garima Jain Feb 18 '19 at 8:47
  • $\begingroup$ You can never categorically say which one algorithm will do always better. It is best to try out multiple algorithms on your data to see which one of them does best for your data. If you have a lot of categorical variables with high cardinality (number of levels) then it is easier to use catboost because it has in-built capability to encode them. With XgBoost, you will have to encode them manually. Still my recommendation is to try them both on your data. If you have any specific questions about these algorithms then let me know. $\endgroup$ – Garima Jain Feb 18 '19 at 8:51
  • $\begingroup$ Thank you very much Garima for your helpful explanation, I do appreciate it. $\endgroup$ – Nirmine Feb 19 '19 at 17:49
0
$\begingroup$

You could try reducing number of features by assessing their importance. This can be achieved by training XGBoost on the data and then analyzing how each feature split improved gini score. More on this and the code you can find: here . You could get the sense of how many features are enough for the prediction for a small information loss.

Another idea is to perform PCA (Principal component analysis) to extract non-correlated features explaining a certain percentage of the target variable variance. You could then run any classification algorithm.

Improve this answer
$\endgroup$
  • $\begingroup$ Thanks Nemanja, I don't think if I can reduce features. the existing algorithm is programmed as follow: each user plays x number of games and gets a score of each game to predicting his future score I should trace his historic. for example, User1 plays: game 1, game 3, game 7 and User2 plays: game 1, game 5, game 7 game 10. The existing algorithm contains all n games columns (G1, G2,G3....G246). I think I shouldn't reduce features $\endgroup$ – Nirmine Feb 3 '19 at 10:12
0
$\begingroup$

XGBoost or ADABoost can improve for sure your accuracy or F1, especially when you have unbalanced class as they tend to give more weight to those observations that during the training go misclassified.

Improve this answer
$\endgroup$
  • $\begingroup$ Thanks 3nomis, but boosting methods with default parameters improve the metrics values just by 3%, and it takes 20min for that (while the existing algorithm was taking just 1min). and if I want to fit parameters it certainly will take very long hours. I ask if there are some methods to reduce time execution. I think improving accuracy by 3% and spending 20 min doesn't matter !! $\endgroup$ – Nirmine Feb 3 '19 at 9:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.