In other words, I am looking to predict students that will fail out of school before it happens. The data includes socioeconomic status and other related variables.

I have tried an XGB binary classification (both tree and forest), but the problem is that it doesn't penalize severely wrong answers (predicting that a student will be in the bottom 3% in terms of grades, but they're actually A+ students). The result is that the average grades of the predicted students is quite low, but the median grades aren't actually that bad - there are a few extremely bad students that pull down the average but not the median.

I have tried a XGB regression (both tree and forest), but the problem is that I can't get the model to focus on the bottom 3%. It seeks to reduce error for all predictions. I couldn't care less about telling the difference between an A student and a B student, I only need to consistently identify the bottom 3%ile.

I was thinking that perhaps this could lend itself to reinforcement learning instead of supervised, but I know nothing about reinforcement... WOuld it be possible to make a reinforcement model where the goal is to minimize the median grades of the 3% of students predicted? Or are there any other machine learning techniques that would work?

  • $\begingroup$ What kind of data do you have? $\endgroup$ – Guillermo Mosse Jun 17 '20 at 17:08
  • $\begingroup$ 62 features either binary or percentiles. $\endgroup$ – xxanissrxx Jun 17 '20 at 17:47
  • $\begingroup$ Maybe worth looking at modeling “rare events” approach ? I do think suits your problem here ... $\endgroup$ – n1tk Jun 18 '20 at 3:05

Try writing a custom loss function for a regression model!

Keras' neural networks support this, for example. See https://stackoverflow.com/q/43818584/745868

(But many other libraries give support for this as well)

The only special thing about your custom loss function is that it doesn't add up the error of a datapoint if min(pred_y, actual_y) >= THRESHOLD

  • $\begingroup$ Do you have any links that could help me create a loss function? I have no idea how to... eg. for XGB they have to have gradient and hessian which I don't think is possible for median / min functions? $\endgroup$ – xxanissrxx Jun 17 '20 at 17:52
  • $\begingroup$ holy crap I just realized how genius this is. I can just take the normal loss function and then exclude any datapoints that aren't in the bottom 3%ile????????? $\endgroup$ – xxanissrxx Jun 17 '20 at 19:35
  • $\begingroup$ Yes, exactly. Here's an example: heartbeat.fritz.ai/…. Please let me know if you bumped into a wall $\endgroup$ – Guillermo Mosse Jun 17 '20 at 20:00
  • $\begingroup$ I would not exclude everything not at the bottom 3%. You also need to worry about mislabelling an A student as a bottom 3% one ;) $\endgroup$ – Guillermo Mosse Jun 17 '20 at 20:01
  • $\begingroup$ Ahhh ok understood. Is there a way to see the default objective function codes so I can start there and add my threshold? $\endgroup$ – xxanissrxx Jun 17 '20 at 20:37

Writing a custom loss function could be handy, but it may be simpler to just try to treat this as a class balance problem for your regression model. For starters, just try undersampling all of the higher and medium grades until they're close to balanced with your failing students. Given your number of data points and features you can probably still just throw a random forest at it and do fine.

  • $\begingroup$ This was the first thing I tried, but it didn't work. $\endgroup$ – xxanissrxx Jun 18 '20 at 0:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.