My dataset contains 2000 records with 125 meaningful fields 5 of which are distributed along highly skewed lognormal behavior.

I've found that if I eliminate all records below some threshold of this lognormal behavior (by combining the fields together then filtering for Nth percentile), my model improves in accuracy from ~78% to ~86%, using a highly tuned random forests classifier. This filter is only done after splitting my data into train, test (which is done after SMOTE).

What makes this particularly odd is that that filter improves results across multiple sampling methods.

Is this filtering acceptable behavior? Why might it be resulting in better predictions?

  • 1
    $\begingroup$ Do you also threshold test data ? $\endgroup$ – Elliot Sep 2 '19 at 18:14
  • $\begingroup$ I'm not sure what you mean @Elliot $\endgroup$ – Yaakov Bressler Sep 2 '19 at 18:59
  • $\begingroup$ do you also filter out the test data you have splitted before ? $\endgroup$ – Elliot Sep 2 '19 at 19:09
  • $\begingroup$ No @Elliot, the data is split then the filter is applied to the train set only. A second iteration would start from the main data then resplit then refilter. $\endgroup$ – Yaakov Bressler Sep 2 '19 at 22:42
  • $\begingroup$ Okay, I’ll make an answer. $\endgroup$ – Elliot Sep 2 '19 at 23:06

One flaw in your procedure is the use of SMOTE before splitting in train/test. This should be avoided as you may have synthetic examples in the test data which generation depends on training data and that will be highly close to this data in your feature space (as SMOTE uses Euclidean distance).

Moreover, if most of the minority data belongs to the not-skewed region of your specific variables, these points will be also over sampled and so this reduction in the variables space will produce an overly optimistic performance which does not reflect the real distribution of the data.

| improve this answer | |
  • 1
    $\begingroup$ Thanks! This is really neat. I had a loud "ahhhha" moment just now. $\endgroup$ – Yaakov Bressler Sep 3 '19 at 1:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.