9
Actually, your understanding of a random forest is not 100 percent correct. Variables are sampled per split, not by tree. So every tree has access to all variables.
In general, tree based models are not too strongly affected by highly correlated features. There are no numeric stability issues as with least squares. You can easily add a variable twice ...
3
370 rows, with approx. 900 features is not optimal. I would suggest some dimension reduction. PCA, factor analysis, PLS regression are some alternatives.
You could try a lasso- / elasticnet -regression as well.
Here is a good guide. https://scikit-learn.org/stable/tutorial/machine_learning_map/index.html
3
I am debugging your code and I don't get those results, if I copy paste your code and I run it I get:
from sklearn.metrics import accuracy_score
accuracy_score(rfc.predict(X_test),y_test)
>>>0.488
y_test_hat = rfc.predict(X_test)
100 * sum(y_test == y_test_hat) / y_test.shape[0]
>>> 48.8
apply_model_and_calc_accuracies(rfc)
>>> ...
2
rfc.fit(X, y) should be rfc.fit(X_train, y_train)
You are simply memorizing the entire dataset with RandomForestClassifier.
1
I think you are probably overfitting.
The issue is that while you have performed a train/test split, you are selecting your hyperparameters based on the whole dataset! This way you are feeding information to the model, about the test set, through your hyperparameter selection. To be honest I haven't seen this in such small grid searches, but to be sure you ...
1
The problem seems to be that your pipeline uses a fresh instance of RandomForestRegressor, so your param_grid is using nonexistent variables of the pipeline. There are two choices (I tend to prefer the second):
Use rfr in the pipeline instead of a fresh RandomForestRegressor, and change your parameter_grid accordingly (rfr__n_estimators).
Change ...
1
Shap values is your friend.
model = RandomForestRegressor(max_depth=6, random_state=0, n_estimators=10)
model.fit(X_train, Y_train)
import shap
shap_values = shap.TreeExplainer(model).shap_values(X_train)
shap.summary_plot(shap_values, X_train, plot_type="bar")
There are several other techniques with their own drawbacks. Some are:
Permutation Importance
...
1
Let´s say that the best way to choose is empirical. You run both algorithms in the dataset and check which one has better performance.
It's true that you can do a lot of theoretical analysis but at the end you have to try no matter what. They both use decision trees ensemble so the results should not be too different. By experience gradient boosting tends ...
1
My guess is that either your Failure/Normal class is a lot lesser than the other. As such, for a certain (i.e. nth) fold, there only exists instances of one class. You can try doing oversampling the under-represented class to prevent this, or try doing a stratified K-Fold so that each fold will have occurrences of both classes.
1
They all start from the same assumption: time series forecasting can't be treated as a regression/classification problem. It is time dependent, which means our target y at time t depends on what the value y was at t-1.
Time series forecasting must take into account time dependency, but it doesn't have to be the only source of information. Many complex ...
Only top voted, non community-wiki answers of a minimum length are eligible
