1
$\begingroup$

I train the RF and I sumbit to the data competition
but my train score is so much different then their submit socre
is there way to decrease this gap? Here is my code with RF

%%time
# Our data is already scaled we should split our training and test sets
from sklearn.model_selection import train_test_split
from sklearn.metrics import log_loss
def under_over_test(X,y):
    RFC_METRIC = 'gini'  #metric used for RandomForrestClassifier
    #RFC_METRIC= 'multi_logloss'
    NUM_ESTIMATORS = 100 #number of estimators used for RandomForrestClassifier
    NO_JOBS = 2018 #number of parallel jobs used for RandomForrestClassifier
    RANDOM_STATE = 42
    MAX_ROUNDS = 2000 #lgb iterations
    EARLY_STOP = 50 #lgb early stop 
    OPT_ROUNDS = 1000  #To be adjusted based on best validation rounds
    VERBOSE_EVAL = 50 #Print out metric result
    clf = RandomForestClassifier(n_jobs=NO_JOBS, 
                             random_state=RANDOM_STATE,
                             criterion=RFC_METRIC,
                             n_estimators=NUM_ESTIMATORS,
                             verbose=True
                            ) 

    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

    clf.fit(X_train, y_train)
    clf_probs = clf.predict_proba(X_test)
    score_origin = log_loss(y_test, clf_probs)
$\endgroup$
1
$\begingroup$

In a Kaggle-like competition, the data your submission is evaluated on is not part of what is given to you. It may always produce a different result than what you estimate in your training process.

But, if it's very different, then you have likely over-fit and your model is high variance, so, your test error is probably higher than the training error here. In particular, you don't seem to be using early stopping. Your submission error is also then going to be higher than training error.

You mean your test error is lower than submission error, right? they may be quite different than your test error due to the high model variance.

The submission error on average might be higher than your test error, even when done properly, because one tends to 'overfit' the hyperparameter tuning in any validation process. (But here I don't see hyperparam tuning even?)

A more correct process would be, at least, to try to find the best tuning params with some cross-validation process, then refit the model on all the data you have using those params.

$\endgroup$
3
  • $\begingroup$ the competition is using log-loss fur metrics. I tried to use Randomforest,lgb. lgb has the log-loss metric option. does RF also has it? and also RF has the option for early stop? $\endgroup$ – slowmonk Feb 11 '20 at 4:53
  • $\begingroup$ You are also using log-loss. You're not using early stopping though as you seem to intend to. It doesn't quite exist for random forests. Consider xgboost instead if you want early stopping. However, it's just one way to solve the general problem of overfitting. Really you need to tune your hyperparams to find those that don't overfit yet yield the best held-out test set loss. $\endgroup$ – Sean Owen Feb 11 '20 at 13:35
  • $\begingroup$ my train shape is 500000 x 400 so randomforest is only way I can train. xgboost , lgb is too slow with large dataset. This is why I try to test with RF $\endgroup$ – slowmonk Feb 11 '20 at 14:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.