# when my RF log_loss socre is different then submit score?

I train the RF and I sumbit to the data competition
but my train score is so much different then their submit socre
is there way to decrease this gap? Here is my code with RF

%%time
# Our data is already scaled we should split our training and test sets
from sklearn.model_selection import train_test_split
from sklearn.metrics import log_loss
def under_over_test(X,y):
RFC_METRIC = 'gini'  #metric used for RandomForrestClassifier
#RFC_METRIC= 'multi_logloss'
NUM_ESTIMATORS = 100 #number of estimators used for RandomForrestClassifier
NO_JOBS = 2018 #number of parallel jobs used for RandomForrestClassifier
RANDOM_STATE = 42
MAX_ROUNDS = 2000 #lgb iterations
EARLY_STOP = 50 #lgb early stop
OPT_ROUNDS = 1000  #To be adjusted based on best validation rounds
VERBOSE_EVAL = 50 #Print out metric result
clf = RandomForestClassifier(n_jobs=NO_JOBS,
random_state=RANDOM_STATE,
criterion=RFC_METRIC,
n_estimators=NUM_ESTIMATORS,
verbose=True
)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

clf.fit(X_train, y_train)
clf_probs = clf.predict_proba(X_test)
score_origin = log_loss(y_test, clf_probs)


In a Kaggle-like competition, the data your submission is evaluated on is not part of what is given to you. It may always produce a different result than what you estimate in your training process.

But, if it's very different, then you have likely over-fit and your model is high variance, so, your test error is probably higher than the training error here. In particular, you don't seem to be using early stopping. Your submission error is also then going to be higher than training error.

You mean your test error is lower than submission error, right? they may be quite different than your test error due to the high model variance.

The submission error on average might be higher than your test error, even when done properly, because one tends to 'overfit' the hyperparameter tuning in any validation process. (But here I don't see hyperparam tuning even?)

A more correct process would be, at least, to try to find the best tuning params with some cross-validation process, then refit the model on all the data you have using those params.

• the competition is using log-loss fur metrics. I tried to use Randomforest,lgb. lgb has the log-loss metric option. does RF also has it? and also RF has the option for early stop? – slowmonk Feb 11 '20 at 4:53
• You are also using log-loss. You're not using early stopping though as you seem to intend to. It doesn't quite exist for random forests. Consider xgboost instead if you want early stopping. However, it's just one way to solve the general problem of overfitting. Really you need to tune your hyperparams to find those that don't overfit yet yield the best held-out test set loss. – Sean Owen Feb 11 '20 at 13:35
• my train shape is 500000 x 400 so randomforest is only way I can train. xgboost , lgb is too slow with large dataset. This is why I try to test with RF – slowmonk Feb 11 '20 at 14:15