1
$\begingroup$

I am teaching myself some data science and have started a Kaggle project. I have fitted a random forest classification model (using sci-kit learn) with a few millions rows of data. There are two possible outcomes for each row (0 or 1). When I run it against the test data, I get 0 for every row. This is a practical impossibility, but I am at a loss as to how to diagnose my model and how to move forward. Is this an example of extreme overfitting ? Is it more likely a problem with my training data (or test data) that i am not seeing ?

Should I simply increase the # of estimators? Is it possible that I have misformatted the input files in a subtle way that doesn't cause an error ? I am at a loss as to how to move forward.

Improve this question
$\endgroup$
3
  • 2
    $\begingroup$ Context matters. I would suspect a bug or misunderstanding in your script (e.g. a mistake when constructing features of test data, or using the wrong/untrained model when predicting). If it's a 101 competition, you may be better off asking in the Kaggle forum for that competition, or in the Getting Started forum there. To get answers here, you may need to show some of your code and explain it before someone could offer advice. $\endgroup$
    – Neil Slater
    Aug 8, 2015 at 9:33
  • $\begingroup$ Thank you. I have some things to look at as a result of posting there and I may come back here if I have more questions, The code itself is very simple. I think It's either the formatting of the data or the options I am using that are at issue here $\endgroup$
    – chrisfs
    Aug 8, 2015 at 20:41
  • 1
    $\begingroup$ What happens when you use predict_proba()? If "1" is a rare outcome then it's pretty likely that your predictions are different but all less than 0.5. If that's the case then you can do a number of things, such as downsampling your training set or using the class_weights argument when building your RF. $\endgroup$
    – David
    Aug 10, 2015 at 1:39

2 Answers 2

Reset to default
2
$\begingroup$

Some of the possibilities include the following:

1) The training data has class imbalance.

Solution:

  • Train the model using CV = 5 or 10;

  • Do a log transformation to make the target distribution more normal in nature;

  • Check if you can add a weight variable that can fix the distribution.

2) Testing data is a small sub sample. Since it's Kaggle, I believe the testing data will be standard in nature, so the problem might be in the process that is generating the submission script.

As a beginner, you should use any benchmark script available in the forum to start with, which shouldn't have such issues.

Improve this answer
$\endgroup$
0
$\begingroup$

1) Like 0xF suggested: Please check the distribution of the label you are predicting i.e. number of 0's and 1's. If there's a class imbalance problem and you have assigned equal weight to all instances, this problem is most likely to occur.

2) Assuming, you took care of class imbalance, also, use the function predict_proba() along with predict(). Now, take the ration of prob(1) by prob(0). It doesn't matter in case of divide by zero error as you can replace those numbers by some very large number. Now, adjust the margins of classification according to your convenience to minimize false positives and false negatives.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.