0
$\begingroup$

I am going through my first solo machine learning project and would like to gain some insight into what I am doing wrong/what is going on here as I am a bit stuck.

I have been applying machine learning to the Titanic data set with SKlearn and have been holding out 10% of the training data to calculate the accuracy of my fitted models. I also use K-fold cross valdation with 10 folds to evaluate the model performance and choose hyper-parameters. I have so far applied logistic regression and a linear Kernel SVM and in both cases I get 78-80% accuracy on the K-fold validation sets and when applying the fitted classifiers to my held-back previously unseen testing data. However when I predict on Kaggle's test data and submit my predictions it comes back with values around 76% which is significantly less than I'd expect, and well outside the variance in the accuracy values I get with K-fold cross validation.

A link to the Jupyter notebook where I do this is provided below: http://nbviewer.jupyter.org/github/AshleySetter/Kaggle_Competitions/blob/master/Titanic_project/Titanic_machine_learning_clean.ipynb

Could anyone give me some insight into what is going on here and what I am doing wrong?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

  1. It could be because of the percentage of the different class. Imagine your data is 30% survived and 70% died but in Kaggle's test data this ratio may change, i.e 50%-50%. So your model could not predict kaggle's survived part as well as your test data.

  2. you may impute missing by the mean. if you use test data for calculating mean it could be a cheating for your model.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Both good points, I have a question about the second point. Currently I'm imputing the mean/median from the whole training set (which includes my held-back test data) and using the same imputer for the kaggle test data. Should I impute the mean/median values using only the data I am predicting on. I.e. rather than using the imputer fitted on the training data, make a new imputer and impute on the test data separately? Or should I use the same imputed mean/median I got from the training data on missing values in the test data? I wasn't clear on which is the correct approach. $\endgroup$
    – SomeRandomPhysicist
    Jul 16, 2018 at 10:34
  • $\begingroup$ the intention of using mean and median is finding a value which has minimal effects on prediction. So you need real mean or median which is mean or median of all data[train+test kaggle]. $\endgroup$
    – parvij
    Jul 16, 2018 at 10:50
  • $\begingroup$ Also, you could use two other methods to impute missing: 1. fill missing randomly but with the same distribution of data. 2.predicting miss value $\endgroup$
    – parvij
    Jul 16, 2018 at 10:52
  • $\begingroup$ Stratifying my train/test splitting made a big difference to my test performance, it is now much closer to my Kaggle accuracy for both the logistic regression and SVM. $\endgroup$
    – SomeRandomPhysicist
    Jul 16, 2018 at 11:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.