I am working with a typical regressor problem. There are $6$ features in the dataset that I am concerned with. There are about $800$ data points in my dataset. The features and the predicted values have high non-linear correlation so the features are not useless (as far as I understand). The predicted values have a bimodal distribution so I disregard linear model pretty quickly.

So I have tried 5 different models: random forest, extra trees, AdaBoost, gradient boosting and xgb regressor. The training dataset returns $%64$ accuracy and the test data returns $15%$. Both numbers scare me haha. I try tuning the parameters for the random forest but seems like nothing particularly make a drastic difference.

What should I do next if you guys don't mind giving me a recommendation?

Function to tune the hyperparameters

def hyperparatuning(model, train_features, train_labels, param_grid = {}):
    grid_search = GridSearchCV(estimator = model, param_grid = param_grid, cv = 3, n_jobs = -1, verbose =2)
    grid_search.fit(train_features, train_labels)
    return grid_search.best_estimator_`

Function to evaluate the model

def evaluate(model, test_features, test_labels):
    predictions = model.predict(test_features)
    errors = abs(predictions - test_labels)
    mape = 100*np.mean(errors/test_labels)
    accuracy = 100 - mape
    print('Model Perfomance')
    print('Average Error: {:0.4f} degress. '.format(np.mean(errors)))
    print('Accuracy = {:0.2f}%. '.format(accuracy))
  • $\begingroup$ More of a query: are Ensemble methods some kind of cocktail models. If so, and if there's any distinct pattern in the error terms, you may try Ensemble route. (ref. ESL, pg 605). $\endgroup$ Jul 15 '19 at 6:33
  • $\begingroup$ The first thing to try is improving your validation method - 64% on training dataset and 15% on test implies that your model is severely overtrained. $\endgroup$
    – timchap
    Jul 16 '19 at 9:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.