0
$\begingroup$

I'm currently working on a machine learning project. It's a supervised learning problem. My goal is to predict for given data of an animal(keeping,size,weight,...) ingredients(energy,vitamine etc..). First i have cleaned the data and encoded the categorial features with LabelEncoding. I choose Random Forest as algorithm, because i have read that trees are good for mixed data(categorial and continues). So i have trained the model with several parameters and i have noticed that i get excellent training results but very bad test results. In my opinion this indicates overfitting. The model is learning the noise. So and i know i have two options for that: More data and reducing the complexity of the model. But i have tried PCA, remove some features, changed the hyperparameter(max_depth to 15). But none of these actions helped. I have reduced the max_depth but then i got higher training error but still a massive high test error.

So what could the problem here be? It's clear for me that the test error is always a bit higher than the training error. But in this case it's a very big difference.

from sklearn.model_selection import GridSearchCV
from sklearn.metrics import mean_absolute_error
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LinearRegression
from sklearn.ensemble import RandomForestRegressor
from sklearn.neural_network import MLPRegressor
from sklearn.decomposition import KernelPCA

param_grid = {
    'n_estimators': [i for i in range(50, 500, 50)],
    'max_depth': [i for i in range(5, 20, 5)],
}

estimator = RandomForestRegressor()
X_train, X_test, Y_train, Y_test = train_test_split(X, Y, test_size=0.1, random_state=52)
X_train,scalerX = normalize(X_train)
Y_train,scalerY = normalize(Y_train)
X_test = scalerX.transform(X_test)
Y_test = scalerY.transform(Y_test)



gridModel = GridSearchCV(estimator=estimator,param_grid=param_grid,n_jobs=4,cv=5,scoring='neg_mean_squared_error')
gridModel.fit(X_train,Y_train)


print(gridModel.best_params_)

best_params: {'max_depth': 15, 'n_estimators': 150} when changing the grid to [i for i in range(5, 50, 5)] then best_params: {'max_depth': 30, 'n_estimators': 50}

y_pred_test = gridModel.predict(X_test)
test_r2_score = r2_score(y_pred=y_pred_test,y_true=Y_test)

y_pred_train = gridModel.predict(X_train)
train_r2_score = r2_score(y_pred=y_pred_train,y_true=Y_train)

print("Result Test:",test_r2_score)
print("Result Train:",train_r2_score)

{'max_depth': 15, 'n_estimators': 150} Result Test: -2.952394644421328e+31 Result Train: 0.8043381537451035 {'max_depth': 30, 'n_estimators': 50} Result Test: -7.37835882483847e+30 Result Train: 0.9286384515560636

    scaler = preprocessing.StandardScaler()
    return scaler.fit_transform(x),scaler```
$\endgroup$
9
  • $\begingroup$ Did you look at the predictions of the test set, and see what wrong predictions it made systematically? $\endgroup$
    – lpounng
    Commented May 16 at 2:05
  • $\begingroup$ @Ipounng i have done some feature selection no i have: Result Test: 0.5896563185597888 Result Train: 0.9320758910567134 but some ouputs are strange: for the training set i get: kieselgur MSE: 14333671.149608213 MAE: 768.1156762190809 R2 : 0.8457572926835822 and for the test set i get: kieselgur MSE: 8625282.224728 MAE: 1520.834075886863 R2 : 0.5210651656421542 $\endgroup$ Commented May 16 at 13:26
  • $\begingroup$ You misunderstood - I mean, did you look at each individual prediction made by the model? Which predictions contribute the most to the total loss, and is there anything common about these predictions? $\endgroup$
    – lpounng
    Commented May 17 at 1:13
  • $\begingroup$ Also worth to look at the samples - save a copy of the train and test data, immediately before feeding into the model, as CSV or XLSX files and inspect them in Excel. Check carefully if there is any column differently scaled/transformed. $\endgroup$
    – lpounng
    Commented May 17 at 1:18
  • $\begingroup$ The take-home message is ML is no magic. Don't expect to throw everything in the black box and pray it would work; it never does. If the anything is not right, prepare to dive in and get your hands dirty with the data. Don't just sit there and read the MSE etc., they are useless for debug. $\endgroup$
    – lpounng
    Commented May 17 at 1:20

1 Answer 1

0
$\begingroup$

A few comments.

  1. I might be missing something, but I can't find your normalize() function, it might have a bug. For random forest there is no need to normalize anything anyway.

  2. If you don't have a lot of data, I would use at least 0.3 for the train validation split. You can also try KFold just to be sure.

$\endgroup$
1
  • $\begingroup$ 1. The normailze() function is a standard scaler. I did this because the the target is a multi target. so each y is a vector. and the content of the vector have different units. 2. Is kFold not in grid search when i set cv=5? And would maybe nested cross validation help? $\endgroup$ Commented May 14 at 6:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.