1
$\begingroup$

I have a large dataset which includes 36 variables (in %iles) to describe a student, and then the output is the students grades as a %ile. I am trying to predict, using the 36 variables, whether a student will fail out, ie. be in the bottom 2%ile.

My first attempt was a neural network with binary classification, forcing the model to predict that the student is either in the bottom 2%ile, or not. Having read a few posts on FHarrell.com, I was thinking that a binary logistic regression that outputs the probability of being in the bottom percentiles is better?

Or alternatively, should I just do a non-binary linear regression to directly predict the students %ile? The reason I thought that this wouldn't be as useful for my purposes is because I don't care about making predictions between any of the other percentiles. For example, I don't care about distinguishing between an A and a B student, I am just looking to identify F students. Therefore I would be losing accuracy in the bottom 2%ile and gaining it in the other irrelevant %iles?

Finally, I think that the neural network makes sense given that there is interaction between the variables. But if there is another model that I should consider please let me know.

Improve this question
$\endgroup$
1
  • $\begingroup$ Be cautious with premature dichotomization - by doing so, you're implicitly telling the algorithm that there is no difference between an A student and a D- student, and that both are equally distinct from an F student. $\endgroup$
    – Nuclear Hoagie
    Oct 28, 2021 at 18:17

1 Answer 1

Reset to default
1
$\begingroup$

My first attempt was a neural network with binary classification, forcing the model to predict that the student is either in the bottom 2%ile, or not. Having read a few posts on FHarrell.com, I was thinking that a logistic regression that outputs the probability of being in the bottom percentiles is better?

I would recommend you start with logistic regression as a baseline. Then when you have your baseline I would compare the LR-model with a tree-based ensemble since they tend to work really well on tabular data. I would recommend random forest, it is easy to work with and is not that prone to overfitting. Then if you are feeling fancy you could try gradient boosting with XGBoost or LGBM. They tend to give marginally better results, but takes some more effort to work with.

Or alternatively, should I just do a non-binary linear regression to directly predict the students %ile? The reason I thought that this wouldn't be as useful for my purposes is because I don't care about making predictions between any of the other percentiles. For example, I don't care about distinguishing between an A and a B student, I am just looking to identify F students. Therefore I would be losing accuracy in the bottom 2%ile and gaining it in the other irrelevant %iles?

I agree.

Finally, I think that the neural network makes sense given that there is interaction between the variables.

Neural networks are generally not that good on tabular data. Unbalanced, which your have with a ration of 1:50, is especially hard for NNs. Then they tend to classify everything as the majority class.

Improve this answer
$\endgroup$
4
  • $\begingroup$ OK thank you very much and just to clarify, when you say "I agree." you think that I am correct? That a linear regression would lose accuracy where I want it and hence is not a good idea? $\endgroup$
    – xxanissrxx
    May 29, 2020 at 16:55
  • $\begingroup$ Sorry and one more question, do you recommend a random forest regressor or random forest classifier? $\endgroup$
    – xxanissrxx
    May 29, 2020 at 17:25
  • $\begingroup$ Yes, that is what I am agreeing with. Do binary classification because it is more in line with your objective. $\endgroup$
    – Simon Larsson
    May 29, 2020 at 23:39
  • 1
    $\begingroup$ I also strongly recommend to use PCA, if you use logistic regression. Due to the fact that, there are many variables of student are irrelevant to the percentiles e. g., length , and hair color. I worked with logistic regression a week ago with tabular data containing 60 columns (5 columns after using PCA) , the training accuracy was 84% without PCA, 97% with PCA and tuning some parameters such as batch size and random_state. Please don't forget to standardize the data before the PCA. $\endgroup$
    – user119783
    Jun 29, 2021 at 6:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.