0
$\begingroup$

Suppose I have a binary classifier that predicts if students will fail a test based on independent features.

How can I update this model's prediction as more information regarding a student's test outcome is available to predict its performance (fail or not) on the following tests?

For example, for test #1, the model outputs "fail," and the student indeed fails. I wish to take this new information (the fact that the student failed) as extra information to use while predicting if the student will fail test #2.

Is there an algorithm or technique that allows me to update the model as this new information comes?

I thought of several ways, but none of them seems right.

  1. Having as a feature of my training set the test number (this could work, but I don't know the total amount of tests)
  2. Having as a feature the outcome of the previous test (this one faces the same issue as the previous case)
  3. Implementing online learning and updating the model with each new outcome.
  4. Instead of training such as classifier, learning somehow a decay weight I could use with the last test's outcome to obtain the new one.
$\endgroup$
5
  • $\begingroup$ see online-learning as an example $\endgroup$
    – Nikos M.
    Nov 11, 2021 at 16:47
  • $\begingroup$ Do you have historical data to train your model on? Are you working with a dataset with observations of past students, their test results, and whether or not they failed? Or are you trying to make predictions on a student's outcome without data on students in past classes? Your comment about not knowing the total number of tests indicates that you might not have historical data. If this is the case, it might be worth investigating some unsupervised or semi-supervised techniques. $\endgroup$
    – E. Kenney
    Nov 11, 2021 at 18:06
  • $\begingroup$ Also, are your observations sequential? For example, is student #1 enrolled a semester before student #2 or are they enrolled in the same semester? This could impact the way in which you train your model. $\endgroup$
    – E. Kenney
    Nov 11, 2021 at 18:19
  • $\begingroup$ Do your independent features change over time (i.e. between each test)? $\endgroup$
    – E. Kenney
    Nov 11, 2021 at 20:08
  • $\begingroup$ > Your comment about not knowing the total number of tests indicates that you might not have historical data. Exactly. I don't have historical data. I just have a dataset made of a bunch of features about the student, i.e., "favorite_class" (as in math, science, etc...) and the exam outcome as my label. > Are you working with a dataset with observations of past students, their test results, and whether or not they failed? Yes. Exactly this. We can assume the students I have on my dataset are not the current students I'll use the model with. $\endgroup$ Nov 12, 2021 at 8:15

1 Answer 1

4
$\begingroup$

First, I will address your thoughts then I will offer some suggestions.

Having as a feature of my training set the test number (this could work, but I don't know the total amount of tests)

I have interpreted this as the creation of a "tests" column with a list of tests the individual has taken (i.e. [1, 2, 3])? We can think of this as a binary indicator of whether or not someone has taken a test. Given that you will only see an indicator for later tests when an individual has completed all the previous tests, the variables will be correlated. Additionally, it does not provide information on an individual's test performance. Furthermore, multiple measurements of a single individual, which violates assumptions behind some approaches.

Having as a feature the outcome of the previous test (this one faces the same issue as the previous case)

I will address this in predicting multiple events.

Implementing online learning and updating the model with each new outcome.

It's worth considering how you will use your trained model. Are you making predictions on a large group of students from a variety of classes every day? Or are you making predictions on a set of students in one class before each test? The model will only learn at each step that you update it. It will get better as you use it more, which means you might need a large amount of data for an online approach. It may be worth considering how often you are planning on updating it and if it makes more sense to retrain a static model once a year (after all of the students take exam #1, for example).

Instead of training such as classifier, learning somehow a decay weight I could use with the last test's outcome to obtain the new one.

Weight decay is used to avoid over fitting. This could be a component of an approach you choose, but I don't think it addresses your question. Perhaps you meant to say Survival Analysis. A reasonable application for this might be predicting student drop out rate.

Predicting Multiple Events

It sounds like you are trying to predict the outcome of multiple events: the outcome of test #1, the outcome of test #2, .... the outcome of test n. I would interpret your labels as different types of events, since the content of test #2 is likely different from the content of test #3.

A naive way to accomplish this might be to train a separate binary classifier for each event (test outcome). The features that you input can be the test(s) results prior. I believe this is what you meant by "Having as a feature the outcome of the previous test". I asked a clarifying question about your dataset that should provide more context as to whether this is reasonable.

X features y predictor
student features test 1 outcome
student features, test 1 outcome test 2 outcome
student features, test 1 outcome,... test n-1 outcome test n outcome

Predicting with Repeated Observations (for the same event)

Your data seems to consist of repeated measurements of multiple individuals, where more variables (i.e. test results) are measured at each time point. If we assume each of your events are the same type of event (i.e. someone retaking the SAT), you might be able to model this as a Repeated Measures Model.

Other Resources

If you are looking for other features to measure that might indicate student success, here's an article on Predicting Student Outcomes. This article analyzes their network of friends. Depending on your dataset, collaborative filtering might be a reasonable approach. These researchers applied a similar technique to their study on student performance. It might be beneficial to explore the why behind your prediction and consider structuring this as a regression problem where you try to predict the percentage a student might score in a class instead of a binary classification problem.

Finally, if you're interested in exploring more information related to the question of your header, "an algorithm that updates predictions after new information is available", you can look into:

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.