- How can I make sure that the initial model trained on a small dataset will not suffer from overfitting before applying the active learning sampling techniques? because I will use this model to select new unlabeled samples.
2 Answers
$\begingroup$
$\endgroup$
I'm not totally sure about my answer so please take it with a grain of salt.
I think you shouldn't worry too much about the initial model being overfitted:
- This is likely to happen since the initial dataset is small, so the model might have no choice other than to capture patterns that happen by chance.
- The process of active learning is intended to "correct" the initial model progressively. This is not only about the model capturing new details, it can also be about the model re-evaluating previous patterns based on the data.
So my intuition would be to just let the model overfit a bit if it has to. However, if the model overfits a lot and/or is too complex, it means that it will require a lot (possibly too many) instances to be labeled. Depending on the context, this could be a more serious problem: the initial model should be decent enough for the active learning process not to need the labeling of many/all instances.
$\begingroup$
$\endgroup$
2
You should test your model on a validation data set: if the validation score is correct, there is no overfitting.
If you want to ensure that there is no overfitting at all, you will want to test your model on several validation data sets.
answered Jun 24, 2021 at 20:39
-
$\begingroup$ Thank you. That works if the model is simple. But the training dataset is small and the model is complex such as ResNetResNet-50. $\endgroup$– user119783Commented Jun 24, 2021 at 20:43
-
$\begingroup$ In that case, you can combine 2 or 3 different complex models. Perhaps combining with simple models could be interesting, but I would need more information to advice other solutions. $\endgroup$ Commented Jun 24, 2021 at 21:04