1
$\begingroup$

I trained the data this way :
There are four classes, the data distributed evenly (same amount of labels).

  1. Used min_max_scaler
  2. Used train_test_split(X,y,test_size=0.3,random_state=42,stratify=y)
  3. Ran GradientBoostingClassifier - once with n_estimators=32, and once with n_estimators=500 on the training data
  4. Used predict on the test data
  5. Got accuracy=0.94 on n_estimators=32 and accuracy=1 on n_estimators=500.
    Precision and recall from classification report is also 1 for all class

Seems fishy but I can't figure out why... what am I doing wrong ?

$\endgroup$
  • $\begingroup$ Are the observations independent? $\endgroup$ – Michael M Dec 9 '18 at 15:43
  • 1
    $\begingroup$ Have you tried with cross-validation? Maybe your seed creates an unusually perfect split. $\endgroup$ – Skiddles Dec 9 '18 at 17:11
  • $\begingroup$ Sorry to ask the obvious, but is your label being used in the inputs? $\endgroup$ – Skiddles Dec 9 '18 at 17:13
  • $\begingroup$ @MichaelM yes, each example is independent of the others $\endgroup$ – M.F Dec 10 '18 at 7:05
  • $\begingroup$ Since you have split your data into training set and test set, it would be helpful to report both the training accuracy and test accuracy. You may also want to report the results with different train_test_split (vary your random_state) in step 2 to see if your observations are consistent across different splits. $\endgroup$ – user12075 Dec 23 '18 at 8:17
1
$\begingroup$

Depending on your data, you may be overfitting, however that isn't necessarily the definitive answer.

Gradient boosted trees are a powerful algorithm and for a while performed as state-of-the-art. If your data happen to represent the target value in a systematic way that you haven't uncovered yet, it's likely that with 500 estimating trees the algorithm found a perfect solution. It's not unheard of.

On the other hand, I don't know much about your data. How many samples do you have? 100? 100,000? The former will be much easier to perfectly model. The latter may also be predictable (albeit less likely) if the variance between classes is predictable. The number of features may also play a role, and the significance of each feature.

As suggested in the comments, Cross Validation may help you discover what's going on here. I highly suggest reading the paper I linked above to see an example of rigorous CV. Carefully read what they did to see how you can model your own CV setup.

You might consider checking out the feature importance returned by your classifier. If one feature is significantly important, it might indicate a close correlation between that feature and the target variable (which should indicate that you need to take a close look at that feature).

| improve this answer | |
$\endgroup$
0
$\begingroup$

Theoretically, you are not allowed to use the min_max_scaler on all the data, because than your test set is not independent anymore. But this is only a problem if you have a very small dataset.

| improve this answer | |
$\endgroup$
0
$\begingroup$

Options :

  1. You problem is too easy to fix, your classes are separable probably without the need of ML

  2. You're leaking information from your target to your test set hence the high accuracy, try it on a different test set scaling your whole data ( and all other estimation operations should be learnt and applied train data and then propagated to the test set.

  3. You're severly overfitting ( the 32 is not so many estimators ) what makes me think that you don't have much data.

| improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.