4
$\begingroup$

Context: I have some data to fit a random forest classifier (binary output) with 1 being a very rare event. In particular, in my training set, there are only 614 1's out of 29400 points. I am using sklearn RandomForestClassifier.

I am setting class_weight = balanced to prevent the model from simply predicting 0 to every case. And it is working great!

However, there is also a small group within the 0 class (maybe only 20 - 30 cases) (Edit: actually about 300) that I want my model to capture. I believe because of the sampling natures when building trees, these class are not chosen very often. Is there a known way to solve this problem?

My thought:

  1. Add an additional filter after the RF. Trouble is, it's hard to find some easy categorization methods for these 20 -30 negative cases.

  2. Force the RF to include these samples when building the trees. Hence this post...

Thanks.

Improve this question
$\endgroup$
3
$\begingroup$

I would suggest that you drop those 30 events and all 1's from your dataset (i presume you know which 30 they are). Then randomly select 584 samples from the remaining dataset and the stick it back together. This would give you a full dataset equally weighted that contains all 30 for definite.

See the following pseudo code (it won't run because i don't know how your identifying these 30 cases so i'm generalising)

data = import()

ones = data.select('1')

interesting = data.select('something to select the 30 data points')

data.drop('1' and 'something to select the 30 data points')

randomList = randomListGen()

selected = data.select(randomList)

joined = join(selected, ones, interesting)

As i said before this won't run on anything, if you give the language you're working in and a minimum working example then i might be able to produce something more solid.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I am running in python. Let me try this idea and get back to you. $\endgroup$ – Matt Keller Aug 14 '19 at 16:24
  • $\begingroup$ Sorry my estimates were wrong. There are actually 200 - 300 such events instead of 20 to 30. I don't think including all 300 in a sample of 600 is wise... any suggestions? $\endgroup$ – Matt Keller Aug 15 '19 at 9:27
  • 2
    $\begingroup$ In what way do the samples differ from the rest? With that many it might be worth having a pre-processer that decides if its a special case and if it is then it runs a different algorithim. In particularly how do you know that they are different? $\endgroup$ – Tasty213 Aug 15 '19 at 9:49
  • $\begingroup$ Tbh I can't really tell. If I could, I would run the filter, as described in my thought 1. The 0/1 values in the training sets are entirely based on my own intuitive judgement, and I was hoping ML could replicate this process to some extent. $\endgroup$ – Matt Keller Aug 15 '19 at 11:11
  • 1
    $\begingroup$ I think you may be better off using the Kmeans minibatch algorithim scikit-learn.org/stable/modules/… . I've never used it personally so might not be able to help much (would advise opening new topic for questions on it). In future this flowchart is pretty much my starting point to find which algorithim to use scikit-learn.org/stable/tutorial/machine_learning_map/… $\endgroup$ – Tasty213 Aug 15 '19 at 12:26
1
$\begingroup$

If you can find a column that has a value to select on, you can use stratify in the train_test_split function. Stratify will try to select an equal number of cases of each value, similar to what you are using. You won't capture all of them, just an equal sample of value vs non-value, but this would be a better approach than forcing a non-random sampling on your model.

train_test_split(..., stratify=df['column'])

after this split, the train and the test splits will have the same ratio of values for the column you selected that the original dataframe has.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.