My dataset contains 2000 records with 125 meaningful fields 5 of which are distributed along highly skewed lognormal behavior.
I've found that if I eliminate all records below some threshold of this lognormal behavior (by combining the fields together then filtering for
Nth percentile), my model improves in accuracy from ~78% to ~86%, using a highly tuned random forests classifier. This filter is only done after splitting my data into train, test (which is done after SMOTE).
What makes this particularly odd is that that filter improves results across multiple sampling methods.
Is this filtering acceptable behavior? Why might it be resulting in better predictions?