And one last thing, is my result on X_train indicative that my features are informative enough to learn the target? or is the R² train score somehow biased?
High scoring fits on training data does not necessarily indicate that your features are informative enough to learn the target in a general fashion. Only your cross validation scores can do so.
Note : In this code i'm using 10 folds for CV. Used 3 folds gave me a better result on CV, if someone can also explain that , it would be great.
I wouldn't expect wildly significant differences so it depends on how much better they were, but remember that you're randomly selecting a subset of data for each fold. It's plausible that when you ran it for 3 folds the model was coincidentally better able to learn from the selected training sets to predict the validation sets. Generally speaking, the more folds you use the more confident you should be in the reliability of the score.
My question: is this considered an overwhelming overfitting problem? or is it mild? and should I do more feature engineering or tune hyperparameters more? ( used GridSearchCV for the current hyperparameters )
I'm no professional, but I would consider that to be severely overfit. I would in the first instance return to hyperparameter tuning to try and bring them in line. I'll also perform the obligatory plug of RandomizedSearchCV at this point: http://www.jmlr.org/papers/volume13/bergstra12a/bergstra12a.pdf
EDIT:
Since you've asked for further details:
1) Does a high score against training data indicate that the features are informative enough to learn the target?
Consider the following example:
import numpy as np
from xgboost import XGBRegressor
x = np.random.rand(5000, 12)
y = np.random.randint(0, 200, 5000)
rfr = XGBRegressor(colsample_bytree=0.6,gamma=0.3,learning_rate=0.1,max_depth=8,min_child_weight=3,subsample=0.9,n_estimators=1000,eval_metric='rmse')
rfr.fit(x, y)
rfr.score(x, y)
Out[26]: 0.999918392694166
Entirely random targets trained against entirely random inputs, and still scoring almost perfectly. That's a completely useless model; validating against your training data does not allow you to determine that your features are informative enough to allow you to predict the target against data that your model has not seen. Only cross-validation against unseen data can do this.
2) Used 3 folds gave me a better result on CV, if someone can also explain that , it would be great.
The RNG gods smiled upon you. Again, consider an example:
from xgboost import XGBClassifier, XGBRegressor
import pandas as pd
from sklearn.model_selection import KFold
df = pd.read_csv(r'https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv', index_col=None)
x = df.drop('species', axis=1)
y = df.species
xgb = XGBClassifier(max_depth=3)
kf = KFold(n_splits=10, shuffle=True)
for train_idx, test_idx in kf.split(x):
xgb.fit(x.iloc[train_idx], y.iloc[train_idx])
print(xgb.score(x.iloc[test_idx], y.iloc[test_idx]))
Out[17]: 0.9333333333333333
0.9333333333333333
0.9333333333333333
1.0
0.8666666666666667
1.0
1.0
0.9333333333333333
0.8666666666666667
1.0
That code splits the data 10 times and then trains and scores the model on each selected slice. As you can see, there's a 13.4% difference between the highest and worst scoring slices.
Your 3-Fold run coincidentally gave you 3 'good' folds. The more Folds you have, the more representative of the true result your cross-validation is.
3) is this considered an overwhelming overfitting problem? or is it mild? and should I do more feature engineering or tune hyperparameters more? ( used GridSearchCV for the current hyperparameters )
Well, whether or not it's overwhelming is something of a matter of opinion. In my mind however a loss against the validation set that is 300% the training score is very severely overfit indeed, but if your validation RMSE is still within the margin of error you're willing to accept then I suppose you could go ahead and use it anyway. It's really your call.
To reduce the overfitting, you need to tune your hyperparameters better. Reducing max_depth and increasing min_samples_split is my usual go-to with trees. If your revised model (exhibiting either no overfitting or at least significantly reduced overfitting) then has a cross-validation score that is too low for you, you should return at that point to feature engineering. We can highlight the effect of hyperparameters on overfitting quite easily by plotting the effect of tweaking them:
from xgboost import XGBClassifier, XGBRegressor
import pandas as pd
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt
from sklearn.preprocessing import OneHotEncoder
df = pd.read_csv(r'https://raw.githubusercontent.com/mwaskom/seaborn-data/master/diamonds.csv', index_col=None)
ohe = OneHotEncoder(sparse=False)
x = pd.DataFrame(ohe.fit_transform(df[['cut', 'color', 'clarity']]))
df.drop(['cut', 'color', 'clarity'], axis=1, inplace=True)
df = df.merge(x, left_index=True, right_index=True)
x = df.drop('price', axis=1)
y = df.price
cv = {}
for i in range(20):
xgb=XGBRegressor(max_depth=i+1)
x_train, x_test, y_train, y_test = train_test_split(x, y)
xgb.fit(x_train, y_train)
cv[i+1] = (xgb.score(x_train, y_train), xgb.score(x_test, y_test))
train = []
test = []
for i in cv.keys():
train.append(cv[i][0])
test.append(cv[i][1])
fig = plt.figure()
plt.plot(train)
plt.plot(test)
plt.legend(['train', 'test'])
plt.xlabel('max_depth')
plt.xlim(0, 20)
plt.ylabel('R^2 Score')
plt.show()
Apologies for the stupid x-axis labels :P. We can see the affect of the max_depth hyperparameter on the model's tendency to overfit easily; anything over 3-ish and the train/test scores begin to diverge.
Hope that those revisions are more helpful to you :)
