In other words, I am looking to predict students that will fail out of school before it happens. The data includes socioeconomic status and other related variables.

I have tried an XGB binary classification (both tree and forest), but the problem is that it doesn't penalize severely wrong answers (predicting that a student will be in the bottom 3% in terms of grades, but they're actually A+ students). The result is that the average grades of the predicted students is quite low, but the median grades aren't actually that bad - there are a few extremely bad students that pull down the average but not the median.

I have tried a XGB regression (both tree and forest), but the problem is that I can't get the model to focus on the bottom 3%. It seeks to reduce error for all predictions. I couldn't care less about telling the difference between an A student and a B student, I only need to consistently identify the bottom 3%ile.

I was thinking that perhaps this could lend itself to reinforcement learning instead of supervised, but I know nothing about reinforcement... WOuld it be possible to make a reinforcement model where the goal is to minimize the median grades of the 3% of students predicted? Or are there any other machine learning techniques that would work?

  • $\begingroup$ What kind of data do you have? $\endgroup$ Jun 17, 2020 at 17:08
  • $\begingroup$ 62 features either binary or percentiles. $\endgroup$
    – xxanissrxx
    Jun 17, 2020 at 17:47
  • $\begingroup$ Maybe worth looking at modeling “rare events” approach ? I do think suits your problem here ... $\endgroup$
    – n1tk
    Jun 18, 2020 at 3:05

1 Answer 1


Try writing a custom loss function for a regression model!

Keras' neural networks support this, for example. See https://stackoverflow.com/q/43818584/745868

(But many other libraries give support for this as well)

The only special thing about your custom loss function is that it doesn't add up the error of a datapoint if min(pred_y, actual_y) >= THRESHOLD

  • $\begingroup$ Do you have any links that could help me create a loss function? I have no idea how to... eg. for XGB they have to have gradient and hessian which I don't think is possible for median / min functions? $\endgroup$
    – xxanissrxx
    Jun 17, 2020 at 17:52
  • $\begingroup$ holy crap I just realized how genius this is. I can just take the normal loss function and then exclude any datapoints that aren't in the bottom 3%ile????????? $\endgroup$
    – xxanissrxx
    Jun 17, 2020 at 19:35
  • $\begingroup$ Yes, exactly. Here's an example: heartbeat.fritz.ai/…. Please let me know if you bumped into a wall $\endgroup$ Jun 17, 2020 at 20:00
  • $\begingroup$ I would not exclude everything not at the bottom 3%. You also need to worry about mislabelling an A student as a bottom 3% one ;) $\endgroup$ Jun 17, 2020 at 20:01
  • $\begingroup$ Ahhh ok understood. Is there a way to see the default objective function codes so I can start there and add my threshold? $\endgroup$
    – xxanissrxx
    Jun 17, 2020 at 20:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.