First, I will address your thoughts then I will offer some suggestions.
Having as a feature of my training set the test number (this could work, but I don't know the total amount of tests)
I have interpreted this as the creation of a "tests" column with a list of tests the individual has taken (i.e. [1, 2, 3])? We can think of this as a binary indicator of whether or not someone has taken a test. Given that you will only see an indicator for later tests when an individual has completed all the previous tests, the variables will be correlated. Additionally, it does not provide information on an individual's test performance. Furthermore, multiple measurements of a single individual, which violates assumptions behind some approaches.
Having as a feature the outcome of the previous test (this one faces the same issue as the previous case)
I will address this in predicting multiple events.
Implementing online learning and updating the model with each new outcome.
It's worth considering how you will use your trained model. Are you making predictions on a large group of students from a variety of classes every day? Or are you making predictions on a set of students in one class before each test? The model will only learn at each step that you update it. It will get better as you use it more, which means you might need a large amount of data for an online approach. It may be worth considering how often you are planning on updating it and if it makes more sense to retrain a static model once a year (after all of the students take exam #1, for example).
Instead of training such as classifier, learning somehow a decay weight I could use with the last test's outcome to obtain the new one.
Weight decay is used to avoid over fitting. This could be a component of an approach you choose, but I don't think it addresses your question. Perhaps you meant to say Survival Analysis. A reasonable application for this might be predicting student drop out rate.
Predicting Multiple Events
It sounds like you are trying to predict the outcome of multiple events: the outcome of test #1, the outcome of test #2, .... the outcome of test n. I would interpret your labels as different types of events, since the content of test #2 is likely different from the content of test #3.
A naive way to accomplish this might be to train a separate binary classifier for each event (test outcome). The features that you input can be the test(s) results prior. I believe this is what you meant by "Having as a feature the outcome of the previous test". I asked a clarifying question about your dataset that should provide more context as to whether this is reasonable.
| X features |
y predictor |
| student features |
test 1 outcome |
| student features, test 1 outcome |
test 2 outcome |
| student features, test 1 outcome,... test n-1 outcome |
test n outcome |
Predicting with Repeated Observations (for the same event)
Your data seems to consist of repeated measurements of multiple individuals, where more variables (i.e. test results) are measured at each time point. If we assume each of your events are the same type of event (i.e. someone retaking the SAT), you might be able to model this as a Repeated Measures Model.
Other Resources
If you are looking for other features to measure that might indicate student success, here's an article on Predicting Student Outcomes. This article analyzes their network of friends. Depending on your dataset, collaborative filtering might be a reasonable approach. These researchers applied a similar technique to their study on student performance. It might be beneficial to explore the why behind your prediction and consider structuring this as a regression problem where you try to predict the percentage a student might score in a class instead of a binary classification problem.
Finally, if you're interested in exploring more information related to the question of your header, "an algorithm that updates predictions after new information is available", you can look into:
