I am working on a binary classification problem with imbalanced data. The dataset consists of 85% 'false' labels and 15% 'true' labels. I have about 1500 data points and am training with Sklearn ensemble tree methods and XGBoost. I know tree methods tend to work better with more data points but I'm not sure what sampling methods would help. I'm trying to optimize for more 'true' points to be in the top decile. I tried to oversample the 'true' class but since I'm using a top decile conversion, the top decile get artificially populated with a lot of 'true' points. This gave inconsistent conversion scores when I used stratified k-fold cross validation.
Should I split the datasets by class and apply sample techniques? What are some relevant techniques that I can try? I'm working with Python and any relevant libraries would be helpful. Thanks in advance!