Ben Reiniger
  • Member for 3 years, 6 months
  • Last seen this week
L1 & L2 Regularization in Light GBM
Accepted answer
17 votes

First, note that in logistic regression, using both an L1 and an L2 penalty is common enough to have its own name: ElasticNet. (Perhaps see https://stats.stackexchange.com/q/184029/232706 .) So ...

View answer
What would I prefer - an over-fitted model or a less accurate model?
14 votes

It depends mostly on the problem context. If predictive performance is all you care about, and you believe the test set to be representative of future unseen data, then the first model is better. (...

View answer
Using a random forest, would a RandomForest performance be less if I drop the first or the last tree?
Accepted answer
11 votes

The two slightly-smaller models will perform exactly the same, on average. There is no difference baked in to the different trees: "the last tree will be the best trained" is not true. The ...

View answer
Why use fit when already have fit_transform?
Accepted answer
8 votes

It probably is fairly rare to need to use fit and not instead fit_transform for a sklearn transformer. It nevertheless makes sense to keep the method separate: fitting a transformer is learning ...

View answer
Does scikit-learn have a forward selection/stepwise regression algorithm?
8 votes

As of version 0.24, it does! Announcement, documentation

View answer
Can Boosted Trees predict below the minimum value of the training label?
Accepted answer
8 votes

Yes, gradient boosted trees can make predictions outside the training labels' range. Here's a quick example: from sklearn.datasets import make_classification from sklearn.ensemble import ...

View answer
Decision tree with final decision being a linear regression
8 votes

I think the easiest way to do this would be to have a decision tree where the final decision results in a linear formula. Setting aside whether this is actually easiest/best, this type of model does ...

View answer
How do we standardize arrays with NaN?
7 votes

This is no longer the case; as of sklearn 0.20.0, missing values are ignored in such preprocessors' fit and silently passed along in their transform: https://scikit-learn.org/stable/whats_new/v0.20....

View answer
What happens if at leaf node both classes have same number of samples?
Accepted answer
7 votes

This is an implementation detail, and I wouldn't necessarily rely on this behavior, but presently in sklearn, it will choose the "first" class. The predict method calls for the probability ...

View answer
What is the proper way to use early stopping with cross-validation?
7 votes

I suspect this is a "no free lunch" situation, and the best thing to do is experiment with (subsets) of your data (or ideally, similar data disjoint from your training data) to see how the final model'...

View answer
ROC AUC score is much less than average cross validation score
Accepted answer
7 votes

Your test score is incorrect. The ROC curve needs the probability scores from the model, not the class decisions. So replace y_predicted = grid_clf.predict(X_test) with y_predicted = grid_clf....

View answer
Is There a Way to Re-Calibrate Predicted Probabilities After Using Class Weights?
Accepted answer
7 votes

A more general adjustment for resampling (not just the simple undersampling in your linked paper) exists: Add $\ln\left(\frac{p_1(1-r_1)}{(1-p_1)r_1}\right)$ to the log-odds of each prediction, where ...

View answer
Default value of learning rate in adam optimizer - Keras
Accepted answer
7 votes

Learning rate is a very important hyperparameter, and often requires some experimentation. There are some good Related questions here, make sure to check those out. With too large a learning rate, ...

View answer
How to extract trees in XGBoost?
Accepted answer
6 votes

This is an open feature request (at time of writing): https://github.com/dmlc/xgboost/issues/2175 https://github.com/dmlc/xgboost/issues/3439 There, a very wasteful but working solution is mentioned: ...

View answer
Why decision tree needs categorical variable to be encoded?
6 votes

...why is encoding needed on categorical variables? That isn't true; decision trees can be built on both continuous and categorical features. (Why don't tree ensembles require one-hot-encoding? )...

View answer
Why is there a difference between predicting on Validation set and Test set?
6 votes

The most likely thing is that there has been some concept drift. Since your model is trained on data up through 2018 and tested on 2019, things have changed, and some of these changes your model ...

View answer
Probability calibration is worsening my model performance
Accepted answer
6 votes

The probability calibration is just stacking a logistic or isotonic regression on top of the base classifier. The default is logistic, and since the sigmoid is a strictly increasing function, the ...

View answer
What makes a Tree-Structured Parzen Estimator "tree-structured?"
Accepted answer
6 votes

It means that your hyperparameter space is tree-like: the value chosen for one hyperparameter determines what hyperparameter will be chosen next and what values are available for it. From a HyperOpt ...

View answer
handling missing values for LightGBM model
5 votes

The default behavior allows the missing values to be sent down either branch of a split. Replacing with a negative value that is less than all your data forces the (originally) missing values to take ...

View answer
Several independent variables based on the same underlying data
Accepted answer
5 votes

For predictive power, in general, including both shouldn't be a problem. But there is a lot of nuance here. Foremost, if predictive power isn't all you care about: if you're making statistical ...

View answer
Can ridge regression be used for feature selection?
Accepted answer
5 votes

Unlike lasso, ridge does not have zeroing coefficients as a goal, and you shouldn't expect applying ridge penalty to have this effect. So the answer to your title question is "no." However, in your ...

View answer
Understanding predict_proba from MultiOutputClassifier
5 votes

In the MultiOutputClassifier, you're treating the two outputs as separate classification tasks; from the docs you linked: This strategy consists of fitting one classifier per target. So the two ...

View answer
Trying to figure out which the training set is
4 votes

To clarify the proposed classifier (because I think it depends on some nonstandard notation): classify any point that is in the (finite) training set according to its true label. (Clearly this gives ...

View answer
Using MinMaxScaler on Training Set... Do I need to scale the input for a prediction as well?
Accepted answer
4 votes

Yes. Any preprocessing that you did manually before the model object was trained needs to be applied to prediction data; the model is expecting the inputs in the same format as when you trained it. ...

View answer
Possible harm in standardizing one-hot encoded features
4 votes

With unpenalized linear models, there is no difference. The coefficients will just scale to counteract the new scale of the variables, and the intercept will shift to compensate for the centering. ...

View answer
Logistic regression does cannot converge without poor model performance
Accepted answer
4 votes

I've often had LogisticRegression "not converge" yet be quite stable (meaning the coefficients don't change much between iterations). Maybe there's some multicolinearity that's leading to ...

View answer
Is Label Encoding with arbitrary numbers ever useful at all?
Accepted answer
4 votes

First, I generally agree that encoding unordered categories as consecutive integers is not a great approach: you are adding a ton of additional relationships that aren't present in the data. CART ...

View answer
ExtraTreesRegressor criterion
Accepted answer
4 votes

No, extremely-random trees does still optimize splits. It does only pick one random splitting point for each feature (out of those randomly chosen max_features) but then which feature is actually ...

View answer
Can a decision in a node of a decision tree be based on comparison between 2 columns of the dataset?
Accepted answer
4 votes

Yes, but not in any implementation that I am aware of. The idea is mentioned in Elements of Statistical Learning, near the end of section 9.2.4 under the heading "Linear Combination Splits." But ...

View answer
Make a random forest estimator the exact same of a decision tree
Accepted answer
4 votes

You need to set bootstrap=False in the random forest to disable the subsampling. (I originally commented because I expected there to be more impediments [in addition to your already-coded ...

View answer
1
2 3 4 5
14